DeepAI AI Chat
Log In Sign Up

ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning

by   Weihao Yu, et al.
National University of Singapore

Recent powerful pre-trained language models have achieved remarkable performance on most of the popular datasets for reading comprehension. It is time to introduce more challenging datasets to push the development of this field towards more comprehensive reasoning of text. In this paper, we introduce a new Reading Comprehension dataset requiring logical reasoning (ReClor) extracted from standardized graduate admission examinations. As earlier studies suggest, human-annotated datasets usually contain biases, which are often exploited by models to achieve high accuracy without truly understanding the text. In order to comprehensively evaluate the logical reasoning ability of models on ReClor, we propose to identify biased data points and separate them into EASY set while the rest as HARD set. Empirical results show that state-of-the-art models have an outstanding ability to capture biases contained in the dataset with high accuracy on EASY set. However, they struggle on HARD set with poor performance near that of random guess, indicating more research is needed to essentially enhance the logical reasoning ability of current models.


page 23

page 24

page 25


R3: A Reading Comprehension Benchmark Requiring Reasoning Processes

Existing question answering systems can only predict answers without exp...

DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs

Reading comprehension has recently seen rapid progress, with systems mat...

ELASTIC: Numerical Reasoning with Adaptive Symbolic Compiler

Numerical reasoning over text is a challenging task of Artificial Intell...

Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4

Harnessing logical reasoning ability is a comprehensive natural language...

MRCLens: an MRC Dataset Bias Detection Toolkit

Many recent neural models have shown remarkable empirical results in Mac...

Non-Monotonic Reasoning and Story Comprehension

This paper develops a Reasoning about Actions and Change framework integ...

Logical Tasks for Measuring Extrapolation and Rule Comprehension

Logical reasoning is essential in a variety of human activities. A repre...