Recitation-Augmented Language Models

10/04/2022
by   Zhiqing Sun, et al.
0

We propose a new paradigm to help Large Language Models (LLMs) generate more accurate factual knowledge without retrieving from an external corpus, called RECITation-augmented gEneration (RECITE). Different from retrieval-augmented language models that retrieve relevant documents before generating the outputs, given an input, RECITE first recites one or several relevant passages from LLMs' own memory via sampling, and then produces the final answers. We show that RECITE is a powerful paradigm for knowledge-intensive NLP tasks. Specifically, we show that by utilizing recitation as the intermediate step, a recite-and-answer scheme can achieve new state-of-the-art performance in various closed-book question answering (CBQA) tasks. In experiments, we verify the effectiveness of RECITE on three pre-trained models (PaLM, UL2, and OPT) and three CBQA tasks (Natural Questions, TriviaQA, and HotpotQA).

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/01/2023

Reimagining Retrieval Augmented Language Models for Answering Queries

We present a reality check on large language models and inspect the prom...
research
12/18/2022

Can Retriever-Augmented Language Models Reason? The Blame Game Between the Retriever and the Language Model

The emergence of large pretrained models has enabled language models to ...
research
02/23/2023

Dr ChatGPT, tell me what I want to hear: How prompt knowledge impacts health answer correctness

Generative pre-trained language models (GPLMs) like ChatGPT encode in th...
research
05/11/2023

Active Retrieval Augmented Generation

Despite the remarkable ability of large language models (LMs) to compreh...
research
10/30/2022

An Efficient Memory-Augmented Transformer for Knowledge-Intensive NLP Tasks

Access to external knowledge is essential for many natural language proc...
research
12/16/2021

Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks

Retrieval-augmented generation models have shown state-of-the-art perfor...
research
12/20/2022

Generation-Augmented Query Expansion For Code Retrieval

Pre-trained language models have achieved promising success in code retr...

Please sign up or login with your details

Forgot password? Click here to reset