DeepAI AI Chat
Log In Sign Up

DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs

03/01/2019
by   Dheeru Dua, et al.
0

Reading comprehension has recently seen rapid progress, with systems matching humans on the most popular datasets for the task. However, a large body of work has highlighted the brittleness of these systems, showing that there is much work left to be done. We introduce a new English reading comprehension benchmark, DROP, which requires Discrete Reasoning Over the content of Paragraphs. In this crowdsourced, adversarially-created, 96k-question benchmark, a system must resolve references in a question, perhaps to multiple input positions, and perform discrete operations over them (such as addition, counting, or sorting). These operations require a much more comprehensive understanding of the content of paragraphs than what was necessary for prior datasets. We apply state-of-the-art methods from both the reading comprehension and semantic parsing literature on this dataset and show that the best systems only achieve 32.7 performance is 96.0 comprehension methods with simple numerical reasoning to achieve 47.0

READ FULL TEXT

page 11

page 12

09/16/2021

Numerical reasoning in machine reading comprehension tasks: are we there yet?

Numerical reasoning based machine reading comprehension is a task that i...
10/15/2019

NumNet: Machine Reading Comprehension with Numerical Reasoning

Numerical reasoning, such as addition, subtraction, sorting and counting...
08/18/2021

EviDR: Evidence-Emphasized Discrete Reasoning for Reasoning Machine Reading Comprehension

Reasoning machine reading comprehension (R-MRC) aims to answer complex q...
02/11/2020

ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning

Recent powerful pre-trained language models have achieved remarkable per...
09/16/2020

Question Directed Graph Attention Network for Numerical Reasoning over Text

Numerical reasoning over texts, such as addition, subtraction, sorting a...
04/15/2021

NT5?! Training T5 to Perform Numerical Reasoning

Numerical reasoning over text (NRoT) presents unique challenges that are...
05/04/2020

To Test Machine Comprehension, Start by Defining Comprehension

Many tasks aim to measure machine reading comprehension (MRC), often foc...