Knowledge distillation is a classic approach for transferring knowledge from a powerful teacher model to a smaller student model. While it might be assumed that a stronger teacher model would naturally result in a stronger student model, this is not always the case — especially when the teacher-student gap is…

Neural algorithmic reasoning, proposed in a 2021 paper by DeepMind researchers Veličković and Blundell as “the art of building neural networks that are able to execute algorithmic computation,” has pushed progress in solving algorithmic tasks. …

Automated information retrieval has become a leading paradigm for information access in today’s world, employed every time someone enters a query into a web search engine. Although the information retrieval field has seen rapid progress in recent years thanks to the development of deep neural networks that can retrieve knowledge…

While today’s deep neural networks (DNNs) are driving AI’s deep-learning revolution, determining a DNN’s appropriate complexity remains challenging. If a DNN is too shallow, its predictive performance will suffer; if it is too deep, it will tend to overfit, and its complexity will result in prohibitively high compute costs. In…

While transformer architectures have achieved remarkable success in recent years thanks to their impressive representational power, their quadratic complexity entails a prohibitively high energy consumption that hinders their deployment in many real-life applications, especially on resource-constrained edge devices. A Monash University research team addresses this issue in the new paper…

The Atari57 suite of classic video games is a popular benchmark used in the reinforcement learning (RL) community to test the general competency of RL algorithms. As Syncedreported in 2020, DeepMind researchers created the first deep RL agent — Agent57 — to achieve above-human performance on all 57 games. While…

In recent years, large language models (LLMs) trained on huge corpora have made tremendous progress in the field of natural language processing, resulting in countless successful real-world applications. Broadly speaking, LLMs can be used under two main settings: a text2vec (text-to-vector) setting for natural language understanding tasks; and a text2text…

Behaviour cloning is a flexible class of approaches employed in control or reinforcement learning (RL) settings for transferring behaviours from expert demonstrations — such as those obtained via a human teleoperating the relevant system directly — to a learned student policy. Behavioural cloning however becomes less efficient in settings where…

While today’s large language models have demonstrated an impressive ability to extract and recall massive amounts of factual knowledge from their training data, the question of how this factual knowledge is stored in such models remains under-explored. In the new paper Knowledge Neurons in Pretrained Transformers, a team from Peking…