Possible Stories: Evaluating Situated Commonsense Reasoning under Multiple Possible Scenarios

09/16/2022
by   Mana Ashida, et al.
0

The possible consequences for the same context may vary depending on the situation we refer to. However, current studies in natural language processing do not focus on situated commonsense reasoning under multiple possible scenarios. This study frames this task by asking multiple questions with the same set of possible endings as candidate answers, given a short story text. Our resulting dataset, Possible Stories, consists of more than 4.5K questions over 1.3K story texts in English. We discover that even current strong pretrained language models struggle to answer the questions consistently, highlighting that the highest accuracy in an unsupervised setting (60.2 far behind human accuracy (92.5 we observe that the questions in our dataset contain minimal annotation artifacts in the answer options. In addition, our dataset includes examples that require counterfactual reasoning, as well as those requiring readers' reactions and fictional information, suggesting that our dataset can serve as a challenging testbed for future studies on situated commonsense reasoning.

READ FULL TEXT

page 1

page 8

page 20

page 24

page 25

research
09/20/2019

Teaching Pretrained Models with Commonsense Reasoning: A Preliminary KB-Based Approach

Recently, pretrained language models (e.g., BERT) have achieved great su...
research
04/22/2019

SocialIQA: Commonsense Reasoning about Social Interactions

We introduce SocialIQa, the first large-scale benchmark for commonsense ...
research
03/14/2018

MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge

We introduce a large dataset of narrative texts and questions about thes...
research
10/12/2020

Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning

Abductive and counterfactual reasoning, core abilities of everyday human...
research
09/10/2019

WIQA: A dataset for "What if..." reasoning over procedural text

We introduce WIQA, the first large-scale dataset of "What if..." questio...
research
06/07/2021

PROST: Physical Reasoning of Objects through Space and Time

We present a new probing dataset named PROST: Physical Reasoning about O...
research
04/15/2020

Personality Assessment from Text for Machine Commonsense Reasoning

This article presents PerSense, a framework to estimate human personalit...

Please sign up or login with your details

Forgot password? Click here to reset