An Empirical Study on the Generalization Power of Neural Representations Learned via Visual Guessing Games

01/31/2021
by   Alessandro Suglia, et al.
0

Guessing games are a prototypical instance of the "learning by interacting" paradigm. This work investigates how well an artificial agent can benefit from playing guessing games when later asked to perform on novel NLP downstream tasks such as Visual Question Answering (VQA). We propose two ways to exploit playing guessing games: 1) a supervised learning scenario in which the agent learns to mimic successful guessing games and 2) a novel way for an agent to play by itself, called Self-play via Iterated Experience Learning (SPIEL). We evaluate the ability of both procedures to generalize: an in-domain evaluation shows an increased accuracy (+7.79) compared with competitors on the evaluation suite CompGuessWhat?!; a transfer evaluation shows improved performance for VQA on the TDIUC dataset in terms of harmonic average accuracy (+5.31) thanks to more fine-grained object representations learned via SPIEL.

READ FULL TEXT

page 3

page 5

09/19/2016

Graph-Structured Representations for Visual Question Answering

This paper proposes to improve visual question answering (VQA) with stru...
07/29/2018

Visual Analogies between Atari Games for Studying Transfer Learning in RL

In this work, we ask the following question: Can visual analogies, learn...
12/30/2021

VisQA: Quantifying Information Visualisation Recallability via Question Answering

Despite its importance for assessing the effectiveness of communicating ...
01/29/2018

Game of Sketches: Deep Recurrent Models of Pictionary-style Word Guessing

The ability of intelligent agents to play games in human-like fashion is...
08/24/2022

UniCon: Unidirectional Split Learning with Contrastive Loss for Visual Question Answering

Visual question answering (VQA) that leverages multi-modality data has a...
08/27/2019

A Data-Efficient Deep Learning Approach for Deployable Multimodal Social Robots

The deep supervised and reinforcement learning paradigms (among others) ...
12/17/2019

Artificial Agents Learn Flexible Visual Representations by Playing a Hiding Game

The ubiquity of embodied gameplay, observed in a wide variety of animal ...