An Efficient Memory-Augmented Transformer for Knowledge-Intensive NLP Tasks

10/30/2022
by   Yuxiang WU, et al.
0

Access to external knowledge is essential for many natural language processing tasks, such as question answering and dialogue. Existing methods often rely on a parametric model that stores knowledge in its parameters, or use a retrieval-augmented model that has access to an external knowledge source. Parametric and retrieval-augmented models have complementary strengths in terms of computational efficiency and predictive accuracy. To combine the strength of both approaches, we propose the Efficient Memory-Augmented Transformer (EMAT) – it encodes external knowledge into a key-value memory and exploits the fast maximum inner product search for memory querying. We also introduce pre-training tasks that allow EMAT to encode informative key-value representations, and to learn an implicit strategy to integrate multiple memory slots into the transformer. Experiments on various knowledge-intensive tasks such as question answering and dialogue datasets show that, simply augmenting parametric models (T5-base) using our method produces more accurate results (e.g., 25.8 -> 44.3 EM on NQ) while retaining a high throughput (e.g., 1000 queries/s on NQ). Compared to retrieval-augmented models, EMAT runs substantially faster across the board and produces more accurate results on WoW and ELI5. Our code and datasets are available at https://github. com/uclnlp/EMAT.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/04/2022

Recitation-Augmented Language Models

We propose a new paradigm to help Large Language Models (LLMs) generate ...
research
06/01/2023

Reimagining Retrieval Augmented Language Models for Answering Queries

We present a reality check on large language models and inspect the prom...
research
06/02/2021

Efficient Passage Retrieval with Hashing for Open-domain Question Answering

Most state-of-the-art open-domain question answering systems use a neura...
research
12/10/2022

REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory

In this paper, we propose an end-to-end Retrieval-Augmented Visual Langu...
research
06/30/2020

Correction of Faulty Background Knowledge based on Condition Aware and Revise Transformer for Question Answering

The study of question answering has received increasing attention in rec...
research
02/22/2022

Retrieval Augmented Classification for Long-Tail Visual Recognition

We introduce Retrieval Augmented Classification (RAC), a generic approac...
research
11/02/2022

Passage-Mask: A Learnable Regularization Strategy for Retriever-Reader Models

Retriever-reader models achieve competitive performance across many diff...

Please sign up or login with your details

Forgot password? Click here to reset