Few-shot Learning with Retrieval Augmented Language Models

08/05/2022
by   Gautier Izacard, et al.
10

Large language models have shown impressive few-shot results on a wide range of tasks. However, when knowledge is key for such results, as is the case for tasks such as question answering and fact checking, massive parameter counts to store knowledge seem to be needed. Retrieval augmented models are known to excel at knowledge intensive tasks without the need for as many parameters, but it is unclear whether they work in few-shot settings. In this work we present Atlas, a carefully designed and pre-trained retrieval augmented language model able to learn knowledge intensive tasks with very few training examples. We perform evaluations on a wide range of tasks, including MMLU, KILT and NaturalQuestions, and study the impact of the content of the document index, showing that it can easily be updated. Notably, Atlas reaches over 42 on Natural Questions using only 64 examples, outperforming a 540B parameters model by 3

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/24/2021

Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models

Prompting language models (LMs) with training examples and task descript...
research
05/18/2023

The Web Can Be Your Oyster for Improving Large Language Models

Large language models (LLMs) encode a large amount of world knowledge. H...
research
05/30/2022

Prompting ELECTRA: Few-Shot Learning with Discriminative Pre-Trained Models

Pre-trained masked language models successfully perform few-shot learnin...
research
08/21/2023

RaLLe: A Framework for Developing and Evaluating Retrieval-Augmented Large Language Models

Retrieval-augmented large language models (R-LLMs) combine pre-trained l...
research
06/30/2023

Meta-training with Demonstration Retrieval for Efficient Few-shot Learning

Large language models show impressive results on few-shot NLP tasks. How...
research
08/01/2023

Retrieval Augmented Generation and Representative Vector Summarization for large unstructured textual data in Medical Education

Large Language Models are increasingly being used for various tasks incl...
research
01/31/2023

Differentiable Entailment for Parameter Efficient Few Shot Learning

Few-shot learning allows pre-trained language models to adapt to downstr...

Please sign up or login with your details

Forgot password? Click here to reset