DeepAI AI Chat
Log In Sign Up

Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets

by   Saku Sugawara, et al.
National Institute of Informatics
Tohoku University
The University of Tokyo

Existing analysis work in machine reading comprehension (MRC) is largely concerned with evaluating the capabilities of systems. However, the capabilities of datasets are not assessed for benchmarking language understanding precisely. We propose a semi-automated, ablation-based methodology for this challenge; By checking whether questions can be solved even after removing features associated with a skill requisite for language understanding, we evaluate to what degree the questions do not require the skill. Experiments on 10 datasets (e.g., CoQA, SQuAD v2.0, and RACE) with a strong baseline model show that, for example, the relative scores of a baseline model provided with content words only and with shuffled sentence words in the context are on average 89.2 These results suggest that most of the questions already answered correctly by the model do not necessarily require grammatical and complex reasoning. For precise benchmarking, MRC datasets will need to take extra care in their design to ensure that questions can correctly evaluate the intended skills.


What Makes Reading Comprehension Questions Easier?

A challenge in creating a dataset for machine reading comprehension (MRC...

SkillQG: Learning to Generate Question for Reading Comprehension Assessment

We present : a question generation framework with controllable comprehen...

Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning

Machine comprehension of texts longer than a single sentence often requi...

Discrete Reasoning Templates for Natural Language Understanding

Reasoning about information from multiple parts of a passage to derive a...

Benchmarking Robustness of Machine Reading Comprehension Models

Machine Reading Comprehension (MRC) is an important testbed for evaluati...

A Survey on Measuring and Mitigating Reasoning Shortcuts in Machine Reading Comprehension

The issue of shortcut learning is widely known in NLP and has been an im...

DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehension

We propose DuoRC, a novel dataset for Reading Comprehension (RC) that mo...