Characterizing Datasets for Social Visual Question Answering, and the New TinySocial Dataset

10/08/2020
by   Zhanwen Chen, et al.
0

Modern social intelligence includes the ability to watch videos and answer questions about social and theory-of-mind-related content, e.g., for a scene in Harry Potter, "Is the father really upset about the boys flying the car?" Social visual question answering (social VQA) is emerging as a valuable methodology for studying social reasoning in both humans (e.g., children with autism) and AI agents. However, this problem space spans enormous variations in both videos and questions. We discuss methods for creating and characterizing social VQA datasets, including 1) crowdsourcing versus in-house authoring, including sample comparisons of two new datasets that we created (TinySocial-Crowd and TinySocial-InHouse) and the previously existing Social-IQ dataset; 2) a new rubric for characterizing the difficulty and content of a given video; and 3) a new rubric for characterizing question types. We close by describing how having well-characterized social VQA datasets will enhance the explainability of AI agents and can also inform assessments and educational interventions for people.

READ FULL TEXT
research
04/17/2020

Knowledge-Based Visual Question Answering in Videos

We propose a novel video understanding task by fusing knowledge-based an...
research
10/23/2019

KnowIT VQA: Answering Knowledge-Based Questions about Videos

We propose a novel video understanding task by fusing knowledge-based an...
research
08/29/2016

Visual Question: Predicting If a Crowd Will Agree on the Answer

Visual question answering (VQA) systems are emerging from a desire to em...
research
03/07/2020

PathVQA: 30000+ Questions for Medical Visual Question Answering

Is it possible to develop an "AI Pathologist" to pass the board-certifie...
research
07/28/2023

Context-VQA: Towards Context-Aware and Purposeful Visual Question Answering

Visual question answering (VQA) has the potential to make the Internet m...
research
05/27/2020

Sybil-proof Answer Querying Mechanism

We study a question answering problem on a social network, where a reque...
research
11/07/2022

CRIPP-VQA: Counterfactual Reasoning about Implicit Physical Properties via Video Question Answering

Videos often capture objects, their visible properties, their motion, an...

Please sign up or login with your details

Forgot password? Click here to reset