FiD-Light: Efficient and Effective Retrieval-Augmented Text Generation

09/28/2022
by   Sebastian Hofstätter, et al.
0

Retrieval-augmented generation models offer many benefits over standalone language models: besides a textual answer to a given query they provide provenance items retrieved from an updateable knowledge base. However, they are also more complex systems and need to handle long inputs. In this work, we introduce FiD-Light to strongly increase the efficiency of the state-of-the-art retrieval-augmented FiD model, while maintaining the same level of effectiveness. Our FiD-Light model constrains the information flow from the encoder (which encodes passages separately) to the decoder (using concatenated encoded representations). Furthermore, we adapt FiD-Light with re-ranking capabilities through textual source pointers, to improve the top-ranked provenance precision. Our experiments on a diverse set of seven knowledge intensive tasks (KILT) show FiD-Light consistently improves the Pareto frontier between query latency and effectiveness. FiD-Light with source pointing sets substantial new state-of-the-art results on six KILT tasks for combined text generation and provenance retrieval evaluation, while maintaining reasonable efficiency.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/07/2022

Multi-Task Retrieval-Augmented Text Generation with Relevance Sampling

This paper studies multi-task training of retrieval-augmented generation...
research
02/02/2022

A Survey on Retrieval-Augmented Text Generation

Recently, retrieval-augmented text generation attracted increasing atten...
research
12/20/2022

Generation-Augmented Query Expansion For Code Retrieval

Pre-trained language models have achieved promising success in code retr...
research
04/13/2023

Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study

Large decoder-only language models (LMs) can be largely improved in term...
research
05/23/2023

FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation

Evaluating the factuality of long-form text generated by large language ...
research
08/01/2023

Retrieval Augmented Generation and Representative Vector Summarization for large unstructured textual data in Medical Education

Large Language Models are increasingly being used for various tasks incl...
research
09/12/2023

Text Encoders Lack Knowledge: Leveraging Generative LLMs for Domain-Specific Semantic Textual Similarity

Amidst the sharp rise in the evaluation of large language models (LLMs) ...

Please sign up or login with your details

Forgot password? Click here to reset