Visual Question: Predicting If a Crowd Will Agree on the Answer

08/29/2016
by   Danna Gurari, et al.
0

Visual question answering (VQA) systems are emerging from a desire to empower users to ask any natural language question about visual content and receive a valid answer in response. However, close examination of the VQA problem reveals an unavoidable, entangled problem that multiple humans may or may not always agree on a single answer to a visual question. We train a model to automatically predict from a visual question whether a crowd would agree on a single answer. We then propose how to exploit this system in a novel application to efficiently allocate human effort to collect answers to visual questions. Specifically, we propose a crowdsourcing system that automatically solicits fewer human responses when answer agreement is expected and more human responses when answer disagreement is expected. Our system improves upon existing crowdsourcing systems, typically eliminating at least 20 effort with no loss to the information collected from the crowd.

READ FULL TEXT

page 1

page 5

page 6

research
08/21/2023

VQA Therapy: Exploring Answer Differences by Visually Grounding Answers

Visual question answering is a task of predicting the answer to a questi...
research
06/04/2021

Visual Question Rewriting for Increasing Response Rate

When a human asks questions online, or when a conversational virtual age...
research
12/12/2022

Design and Evaluation of Crowd-sourcing Platforms Based on Users Confidence Judgments

Crowd-sourcing deals with solving problems by assigning them to a large ...
research
11/30/2019

A Free Lunch in Generating Datasets: Building a VQG and VQA System with Attention and Humans in the Loop

Despite their importance in training artificial intelligence systems, la...
research
10/08/2020

Characterizing Datasets for Social Visual Question Answering, and the New TinySocial Dataset

Modern social intelligence includes the ability to watch videos and answ...
research
10/07/2020

"I'd rather just go to bed": Understanding Indirect Answers

We revisit a pragmatic inference problem in dialog: understanding indire...
research
09/12/2018

The Wisdom of MaSSeS: Majority, Subjectivity, and Semantic Similarity in the Evaluation of VQA

We introduce MASSES, a simple evaluation metric for the task of Visual Q...

Please sign up or login with your details

Forgot password? Click here to reset