DeepAI AI Chat
Log In Sign Up

Numerical reasoning in machine reading comprehension tasks: are we there yet?

by   Hadeel Al-Negheimish, et al.

Numerical reasoning based machine reading comprehension is a task that involves reading comprehension along with using arithmetic operations such as addition, subtraction, sorting, and counting. The DROP benchmark (Dua et al., 2019) is a recent dataset that has inspired the design of NLP models aimed at solving this task. The current standings of these models in the DROP leaderboard, over standard metrics, suggest that the models have achieved near-human performance. However, does this mean that these models have learned to reason? In this paper, we present a controlled study on some of the top-performing model architectures for the task of numerical reasoning. Our observations suggest that the standard metrics are incapable of measuring progress towards such tasks.


page 1

page 2

page 3

page 4


NumNet: Machine Reading Comprehension with Numerical Reasoning

Numerical reasoning, such as addition, subtraction, sorting and counting...

DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs

Reading comprehension has recently seen rapid progress, with systems mat...

Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension

Reading comprehension models have been successfully applied to extractiv...

Question Directed Graph Attention Network for Numerical Reasoning over Text

Numerical reasoning over texts, such as addition, subtraction, sorting a...

A Survey on Measuring and Mitigating Reasoning Shortcuts in Machine Reading Comprehension

The issue of shortcut learning is widely known in NLP and has been an im...

Coreference Reasoning in Machine Reading Comprehension

The ability to reason about multiple references to a given entity is ess...

Quiz Design Task: Helping Teachers Create Quizzes with Automated Question Generation

Question generation (QGen) models are often evaluated with standardized ...