Multi-Task Retrieval-Augmented Text Generation with Relevance Sampling

07/07/2022
by   Sebastian Hofstätter, et al.
0

This paper studies multi-task training of retrieval-augmented generation models for knowledge-intensive tasks. We propose to clean the training set by utilizing a distinct property of knowledge-intensive generation: The connection of query-answer pairs to items in the knowledge base. We filter training examples via a threshold of confidence on the relevance labels, whether a pair is answerable by the knowledge base or not. We train a single Fusion-in-Decoder (FiD) generator on seven combined tasks of the KILT benchmark. The experimental results suggest that our simple yet effective approach substantially improves competitive baselines on two strongly imbalanced tasks; and shows either smaller improvements or no significant regression on the remaining tasks. Furthermore, we demonstrate our multi-task training with relevance label sampling scales well with increased model capacity and achieves state-of-the-art results in five out of seven KILT tasks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/28/2022

FiD-Light: Efficient and Effective Retrieval-Augmented Text Generation

Retrieval-augmented generation models offer many benefits over standalon...
research
12/16/2021

Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks

Retrieval-augmented generation models have shown state-of-the-art perfor...
research
12/20/2022

Generation-Augmented Query Expansion For Code Retrieval

Pre-trained language models have achieved promising success in code retr...
research
11/22/2022

Retrieval-Augmented Multimodal Language Modeling

Recent multimodal models such as DALL-E and CM3 have achieved remarkable...
research
03/15/2017

Ensemble of Neural Classifiers for Scoring Knowledge Base Triples

This paper describes our approach for the triple scoring task at the WSD...
research
12/15/2022

FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference

Fusion-in-Decoder (FiD) is a powerful retrieval-augmented language model...

Please sign up or login with your details

Forgot password? Click here to reset