Contextualized Representations Using Textual Encyclopedic Knowledge

04/24/2020
by   Mandar Joshi, et al.
0

We present a method to represent input texts by contextualizing them jointly with dynamically retrieved textual encyclopedic background knowledge from multiple documents. We apply our method to reading comprehension tasks by encoding questions and passages together with background sentences about the entities they mention. We show that integrating background knowledge from text is effective for tasks focusing on factual reasoning and allows direct reuse of powerful pretrained BERT-style encoders. Moreover, knowledge integration can be further improved with suitable pretraining via a self-supervised masked language model objective over words in background-augmented input text. On TriviaQA, our approach obtains improvements of 1.6 to 3.1 F1 over comparable RoBERTa models which do not integrate background knowledge dynamically. On MRQA, a large collection of diverse QA datasets, we see consistent gains in-domain along with large improvements out-of-domain on BioASQ (2.1 to 4.2 F1), TextbookQA (1.6 to 2.0 F1), and DuoRC (1.1 to 2.0 F1).

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/09/2019

Span Selection Pre-training for Question Answering

BERT (Bidirectional Encoder Representations from Transformers) and relat...
research
12/20/2019

Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model

Recent breakthroughs of pretrained language models have shown the effect...
research
04/21/2019

Probing Prior Knowledge Needed in Challenging Chinese Machine Reading Comprehension

With an ultimate goal of narrowing the gap between human and machine rea...
research
10/29/2017

Simple and Effective Multi-Paragraph Reading Comprehension

We consider the problem of adapting neural paragraph-level question answ...
research
12/13/2021

Roof-BERT: Divide Understanding Labour and Join in Work

Recent work on enhancing BERT-based language representation models with ...
research
01/13/2022

ChartText: Linking Text with Charts in Documents

Recent works show that interactive documents connecting text with visual...

Please sign up or login with your details

Forgot password? Click here to reset