Large-scale language models (LLMs) have achieved impressive performance in automated code generation by bootstrapping human knowledge and learning from extremely large datasets. …

The effective modelling of long-term dependencies enables conditioning new model outputs on previous inputs and is critical when dealing with longer text, audio or video contexts. However, when modelling the long-term dependencies of audio signals, even smaller time scales can yield hundreds of thousands of samples. While transformer architectures have…

The conventional approach for improving the decision-making of deep reinforcement learning (RL) agents is to gradually amortize the useful information they gain from their experiences via gradient descent on training losses. This method however requires building increasingly large models to deal with increasingly complex environments and is difficult to adapt…

Encoder-decoder models have become the preferred approach for a wide range of language-related tasks. Although some common logical functions are shared between different tasks, most contemporary encoder-decoder models are trained end-to-end for a specified task. This specialization increases the compute burden during training and results in less generally interpretable architectures. …

Neural network-based approaches have made significant progress on video compression over the last several years, reaching performance on par with classical codec-based methods. These novel neural approaches however are challenging to implement, as they tend to require complex hand-crafted connections between their many sub-components and struggle when the input data…

Deep neural networks have recently hinted at their potential for processing speech in a manner more like the human brain and generating activations similar to those of the brain in response to the same inputs. …

As AI systems increasingly move from the cloud to devices, identifying suitable neural network backbones for mobile device deployment has become a hot research area. …

Powered by their ever-increasing scale, today’s large language models have shown breakthrough capabilities beyond natural language processing (NLP), in areas such as writing computer code, diagnosing medical conditions and playing competitive games. As the development and deployment of large-scale language models continues, it is important that the AI community understands…

While researchers have traditionally employed Gaussian processes (GP) for specifying prior and posterior distributions over functions, this approach becomes computationally expensive when scaled, is limited by the expressivity of its covariance function, and struggles with adapting a point estimation for the hyperparameters. A research team from the University of Cambridge…