Building on the epoch-making performance of transformer architectures in natural language processing (NLP), the vision transformer (ViT) has emerged as one of the most advanced architectures for computer vision (CV) tasks, demonstrating excellent capabilities in modelling both short- and long-range information compared to conventional convolutional neural network (CNN) approaches. …

The paradigms for natural language processing (NLP) are rapidly evolving — from fully supervised learning to pretraining and fine-tuning and, more recently, pretraining with prompt prediction. The exciting progress and real-world applicability of NLP systems have motivated more AI researchers to explore innovations in this field. In the new paper…

Lossy compression is a data compression approach with a trade-off: it enables high compression but incurs some loss of information, i.e. the compressed data will not be recovered or reconstructed in its exact original state. …

Building a general-purpose unified model that can solve diverse tasks in different modalities while maintaining high performance is a long-standing challenge in the machine learning research community. …

Large-scale language models (LLMs) have achieved impressive performance in automated code generation by bootstrapping human knowledge and learning from extremely large datasets. …

The effective modelling of long-term dependencies enables conditioning new model outputs on previous inputs and is critical when dealing with longer text, audio or video contexts. However, when modelling the long-term dependencies of audio signals, even smaller time scales can yield hundreds of thousands of samples. …

The conventional approach for improving the decision-making of deep reinforcement learning (RL) agents is to gradually amortize the useful information they gain from their experiences via gradient descent on training losses. This method however requires building increasingly large models to deal with increasingly complex environments and is difficult to adapt…

Encoder-decoder models have become the preferred approach for a wide range of language-related tasks. Although some common logical functions are shared between different tasks, most contemporary encoder-decoder models are trained end-to-end for a specified task. This specialization increases the compute burden during training and results in less generally interpretable architectures. …

Neural network-based approaches have made significant progress on video compression over the last several years, reaching performance on par with classical codec-based methods. These novel neural approaches however are challenging to implement, as they tend to require complex hand-crafted connections between their many sub-components and struggle when the input data…