Conversational generative large multimodal models (LMMs) have achieved impressive performance on a wide variety of vision-language tasks. Despite the success of these LMMs in general domain, they normally have worse performance on biomedical field with domain specific biomedical image-text pairs. In an effort to bridge this gap, a new paper…

Deep reinforcement learning (RL) is a trending machine learning algorithm that aims at solving complex decision-making tasks at a human or super-human level performance. …

In recent years, there has been a surge of interest in generative retrieval approaches, which represent a fresh paradigm aiming to transform traditional information retrieval methods. These approaches leverage the power of a single sequence-to-sequence Transformer model to encode and process an entire document corpus. …

Large language models (LLMs) pretrained on massive data are being used in countless real-world applications. However — as computer scientists have known for decades — not all data is equal, and this is also true with regard to the composition of LLM pretraining data, which is typically sourced from publicly…

The 19th-century British philosopher Thomas Carlyle ascribed human progress to a key historical development: “Man is a tool-using animal. Without tools he is nothing, with tools he is all.” While today’s large language models (LLMs) have demonstrated impressive generative and problem-solving capabilities, recent research suggests they could take a similar…

Fine-tuning large-scale pretrained transformers enables them to adapt to and perform better on downstream tasks. While this fine-tuning is crucial for countless real-world applications, fully fine-tuning all model parameters becomes increasingly challenging as models scale to ever-increasing sizes. This has led to the development of parameter-efficient transfer learning (PETL) techniques…

Speech technologies such as automatic speech recognition (ASR) and speech synthesis or text-to-speech (TTS) are playing an increasingly important role in many real-world applications. Contemporary speech technology systems however support only about one hundred languages at best — a tiny fraction of the over 7,000 languages spoken worldwide. A Meta…

The recent rapid rise of large language models (LLMs) has piqued research interest regarding the power and potential of representation models, which are designed to decode and understand data. While contemporary representation models have achieved outstanding performance on unimodal tasks, they typically remain underequipped for handling multimodal tasks. In the…

In April 2022, Google unveiled its 540 billion parameter Pathways Language Model (PaLM), which they developed using a novel Pathways (Barham et al., 2022) approach that enables efficient model training across multiple TPU v4 Pods (in PaLM’s case, 6144 TPU v4 chips). …