DeepAI AI Chat
Log In Sign Up

To Test Machine Comprehension, Start by Defining Comprehension

by   Jesse Dunietz, et al.

Many tasks aim to measure machine reading comprehension (MRC), often focusing on question types presumed to be difficult. Rarely, however, do task designers start by considering what systems should in fact comprehend. In this paper we make two key contributions. First, we argue that existing approaches do not adequately define comprehension; they are too unsystematic about what content is tested. Second, we present a detailed definition of comprehension – a "Template of Understanding" – for a widely useful class of texts, namely short narratives. We then conduct an experiment that strongly suggests existing systems are not up to the task of narrative understanding as we define it.


NumNet: Machine Reading Comprehension with Numerical Reasoning

Numerical reasoning, such as addition, subtraction, sorting and counting...

Spoken SQuAD: A Study of Mitigating the Impact of Speech Recognition Errors on Listening Comprehension

Reading comprehension has been widely studied. One of the most represent...

Reading Comprehension Ability Test-A Turing Test for Reading Comprehension

Reading comprehension is an important ability of human intelligence. Lit...

DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs

Reading comprehension has recently seen rapid progress, with systems mat...

Acyclic Comprehension is equal to Stratified Comprehension

A new criterion of comprehension is defined, initially termed by myself ...

A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task

Enabling a computer to understand a document so that it can answer compr...

Automatic Task Requirements Writing Evaluation via Machine Reading Comprehension

Task requirements (TRs) writing is an important question type in Key Eng...