Augmented Language Models: a Survey

02/15/2023
by   Grégoire Mialon, et al.
0

This survey reviews works in which language models (LMs) are augmented with reasoning skills and the ability to use tools. The former is defined as decomposing a potentially complex task into simpler subtasks while the latter consists in calling external modules such as a code interpreter. LMs can leverage these augmentations separately or in combination via heuristics, or learn to do so from demonstrations. While adhering to a standard missing tokens prediction objective, such augmented LMs can use various, possibly non-parametric external modules to expand their context processing ability, thus departing from the pure language modeling paradigm. We therefore refer to them as Augmented Language Models (ALMs). The missing token objective allows ALMs to learn to reason, use tools, and even act, while still performing standard natural language tasks and even outperforming most regular LMs on several benchmarks. In this work, after reviewing current advance in ALMs, we conclude that this new research direction has the potential to address common limitations of traditional LMs such as interpretability, consistency, and scalability issues.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/15/2023

RAVEN: In-Context Learning with Retrieval Augmented Encoder-Decoder Language Models

In this paper, we investigate the in-context learning ability of retriev...
research
10/11/2022

Decoupled Context Processing for Context Augmented Language Modeling

Language models can be augmented with a context retriever to incorporate...
research
05/25/2022

Training Language Models with Memory Augmentation

Recent work has improved language models remarkably by equipping them wi...
research
05/24/2022

TALM: Tool Augmented Language Models

Transformer based language models (LMs) demonstrate increasing performan...
research
05/19/2023

ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings

Augmenting large language models (LLMs) with external tools has emerged ...
research
02/09/2023

Toolformer: Language Models Can Teach Themselves to Use Tools

Language models (LMs) exhibit remarkable abilities to solve new tasks fr...
research
03/16/2022

Memorizing Transformers

Language models typically need to be trained or finetuned in order to ac...

Please sign up or login with your details

Forgot password? Click here to reset