The empirical success of deep neural networks (DNNs) has inspired the machine learning research community to initiate theoretical studies on DNN aspects such as learning, optimization and generalization. One of the crucial associated research areas is exploring how to design provably time- and sample-efficient learning algorithms for neural networks — a challenge that remains unsolved even in the simplest case of depth-2 feedforward neural networks.

To advance research in this field, a team from Google Research and Northwestern University has introduced a set of polynomial time- and sample-efficient algorithms for learning an unknown depth-2 feedforward neural network with general rectified…

Quantum algorithms for training wide and classical neural networks have become one of the most promising research areas for quantum computer applications. While neural networks have achieved state-of-the-art results across many benchmark tasks, existing quantum neural networks have yet to clearly demonstrate quantum speedups for tasks involving classical datasets. Given deep learning’s ever-rising computational requirements, the use of quantum computers to efficiently train deep neural networks is a research field that could greatly benefit from further exploration.

Motivated by the success of classical deep neural networks (DNNs), a team from the Massachusetts Institute of Technology and Google Quantum AI has…

Neural machine translation (NMT) systems have achieved promising results in recent years, but numerical mistranslation remains a general issue found even in major commercial systems and state-of-the-art research models. A single mistranslated digit can cause severe consequences, especially in systems deployed in the financial and medical fields.

To facilitate the discovery of numerical errors in NMT systems, a research team from the University of Melbourne, Facebook AI, and Twitter Cortex has proposed a black-box test method for assessing and debugging the numerical translation of NMT systems in a systematic manner. …

Although effective uncertainty estimation can be a key consideration in the development of safe and fair artificial intelligence systems, most of today’s large-scale deep learning applications are lacking in this regard.

To accelerate research in this field, a team from DeepMind has proposed epistemic neural networks (ENNs) as an interface for uncertainty modelling in deep learning, and the KL divergence from a target distribution as a precise metric to evaluate ENNs. …

The ability of graph neural networks (GNNs) to deal with graph data structures has made them suitable for real-life applications in social networks, bioinformatics, and navigation and planning problems in robotics. But despite their growing popularity, GNNs are not without their limitations, which include processing efficiency, the high computational complexity problem, and quadratic memory requirements for dense graphs.

To solve these issues, a research team Google Brain, Columbia University and University of Oxford has proposed a new class of graph neural networks, Graph Kernel Attention Transformers (GKATs), which combine graph kernels, attention-based networks with structural priors, and recent transformer architectures…

This is the age of deep neural networks (DNNs), which have proven effective across a wide range of AI applications. While large-scale DNN models learn faster and outperform their slimmer counterparts, these heavy models’ voracious resource appetites have limited their real-world deployment.

Pruning is one of the most popular DNN compression methods, aiming to reduce redundant structures to achieve slimmer architectures and also improve the interpretability of DNN models. Existing pruning methods however are usually heuristic, task-specific, time-consuming, and lack generalization ability.

In the paper Only Train Once: A One-Shot Neural Network Training And Pruning Framework, a research team from…

Neural language models are gaining popularity in real-life creative tasks such as text-adventure games, collaborative slogan writing, and even sports journalism, poetry and novel generation. Most such language models however provide limited interaction support for users, as control that goes beyond simple left-to-right text generation requires explicit training.

To address this limitation, a team from Google Research has proposed Wordcraft, a text editor with a built-in AI-powered creative writing assistant. Wordcraft leverages few-shot learning and the natural affordances of conversation to support a variety of user interactions; and can help with story planning, writing and editing.

The prediction of protein structures from amino acid sequence information alone, known as the “protein folding problem,” has been an important open research question for more than 50 years. In the fall of 2020, DeepMind’s neural network model AlphaFold took a huge leap forward in solving this problem, outperforming some 100 other teams in the Critical Assessment of Structure Prediction (CASP) challenge, regarded as the gold-standard accuracy assessment for protein structure prediction. The success of the novel approach is considered a milestone in protein structure prediction.

This week, the DeepMind paper Highly Accurate Protein Structure Prediction with AlphaFold was published…