End-to-End Training of Neural Retrievers for Open-Domain Question Answering

01/02/2021
by   Devendra Singh Sachan, et al.
0

Recent work on training neural retrievers for open-domain question answering (OpenQA) has employed both supervised and unsupervised approaches. However, it remains unclear how unsupervised and supervised methods can be used most effectively for neural retrievers. In this work, we systematically study retriever pre-training. We first propose an approach of unsupervised pre-training with the Inverse Cloze Task and masked salient spans, followed by supervised finetuning using question-context pairs. This approach leads to absolute gains of 2+ points over the previous best result in the top-20 retrieval accuracy on Natural Questions and TriviaQA datasets. We also explore two approaches for end-to-end supervised training of the reader and retriever components in OpenQA models. In the first approach, the reader considers each retrieved document separately while in the second approach, the reader considers all the retrieved documents together. Our experiments demonstrate the effectiveness of these approaches as we obtain new state-of-the-art results. On the Natural Questions dataset, we obtain a top-20 retrieval accuracy of 84, an improvement of 5 points over the recent DPR model. In addition, we achieve good results on answer extraction, outperforming recent models like REALM and RAG by 3+ points. We further scale up end-to-end training to large models and show consistent gains in performance over smaller models.

READ FULL TEXT
research
10/09/2019

Domain-Relevant Embeddings for Medical Question Similarity

The rate at which medical questions are asked online significantly excee...
research
12/14/2021

You Only Need One Model for Open-domain Question Answering

Recent works for Open-domain Question Answering refer to an external kno...
research
06/09/2021

End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering

We present an end-to-end differentiable training method for retrieval-au...
research
09/23/2022

Variational Open-Domain Question Answering

We introduce the Variational Open-Domain (VOD) framework for end-to-end ...
research
12/05/2022

Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformer

Systems for knowledge-intensive tasks such as open-domain question answe...
research
12/20/2022

To Adapt or to Annotate: Challenges and Interventions for Domain Adaptation in Open-Domain Question Answering

Recent advances in open-domain question answering (ODQA) have demonstrat...
research
10/23/2020

Neural Passage Retrieval with Improved Negative Contrast

In this paper we explore the effects of negative sampling in dual encode...

Please sign up or login with your details

Forgot password? Click here to reset