Evaluating Theory of Mind in Question Answering

08/28/2018
by   Aida Nematzadeh, et al.
0

We propose a new dataset for evaluating question answering models with respect to their capacity to reason about beliefs. Our tasks are inspired by theory-of-mind experiments that examine whether children are able to reason about the beliefs of others, in particular when those beliefs differ from reality. We evaluate a number of recent neural models with memory augmentation. We find that all fail on our tasks, which require keeping track of inconsistent states of the world; moreover, the models' accuracy decreases notably when random sentences are introduced to the tasks at test.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/24/2023

ToMChallenges: A Principle-Guided Dataset and Diverse Evaluation Tasks for Exploring Theory of Mind

Theory of Mind (ToM), the capacity to comprehend the mental states of di...
research
01/17/2023

Memory-Augmented Theory of Mind Network

Social reasoning necessitates the capacity of theory of mind (ToM), the ...
research
04/03/2017

It Takes Two to Tango: Towards Theory of AI's Mind

Theory of Mind is the ability to attribute mental states (beliefs, inten...
research
12/27/2021

Multi-Image Visual Question Answering

While a lot of work has been done on developing models to tackle the pro...
research
07/02/2020

Project PIAF: Building a Native French Question-Answering Dataset

Motivated by the lack of data for non-English languages, in particular f...
research
03/21/2023

A Review on Machine Theory of Mind

Theory of Mind (ToM) is the ability to attribute mental states to others...
research
03/15/2012

Distribution over Beliefs for Memory Bounded Dec-POMDP Planning

We propose a new point-based method for approximate planning in Dec-POMD...

Please sign up or login with your details

Forgot password? Click here to reset