Simple and Efficient ways to Improve REALM

by   Vidhisha Balachandran, et al.

Dense retrieval has been shown to be effective for retrieving relevant documents for Open Domain QA, surpassing popular sparse retrieval methods like BM25. REALM (Guu et al., 2020) is an end-to-end dense retrieval system that relies on MLM based pretraining for improved downstream QA efficiency across multiple datasets. We study the finetuning of REALM on various QA tasks and explore the limits of various hyperparameter and supervision choices. We find that REALM was significantly undertrained when finetuning and simple improvements in the training, supervision, and inference setups can significantly benefit QA results and exceed the performance of other models published post it. Our best model, REALM++, incorporates all the best working findings and achieves significant QA accuracy improvements over baselines ( 5.5 REALM++ matches the performance of large Open Domain QA models which have 3x more parameters demonstrating the efficiency of the setup.


page 1

page 2

page 3

page 4


Open Domain Question Answering over Tables via Dense Retrieval

Recent advances in open-domain QA have led to strong models based on den...

Two-Step Question Retrieval for Open-Domain QA

The retriever-reader pipeline has shown promising performance in open-do...

Dense Hierarchical Retrieval for Open-Domain Question Answering

Dense neural text retrieval has achieved promising results on open-domai...

Learning Dense Representations of Phrases at Scale

Open-domain question answering can be reformulated as a phrase retrieval...

Synthetic Target Domain Supervision for Open Retrieval QA

Neural passage retrieval is a new and promising approach in open retriev...

Augmenting Pre-trained Language Models with QA-Memory for Open-Domain Question Answering

Retrieval augmented language models have recently become the standard fo...

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Language model pretraining has led to significant performance gains but ...