Emergent Communication in a Multi-Modal, Multi-Step Referential Game

05/29/2017
by   Katrina Evtimova, et al.
0

Inspired by previous work on emergent communication in referential games, we propose a novel multi-modal, multi-step referential game, where the sender and receiver have access to distinct modalities of an object, and their information exchange is bidirectional and of arbitrary duration. The multi-modal multi-step setting allows agents to develop an internal communication significantly closer to natural language, in that they share a single set of messages, and that the length of the conversation may vary according to the difficulty of the task. We examine these properties empirically using a dataset consisting of images and textual descriptions of mammals, where the agents are tasked with identifying the correct object. Our experiments indicate that a robust and efficient communication protocol emerges, where gradual information exchange informs better predictions and higher communication bandwidth improves generalization.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/05/2023

Text2Face: A Multi-Modal 3D Face Model

We present the first 3D morphable modelling approach, whereby 3D face sh...
research
09/30/2021

Multi-Modal Sarcasm Detection Based on Contrastive Attention Mechanism

In the past decade, sarcasm detection has been intensively conducted in ...
research
04/20/2019

Multi-modal gated recurrent units for image description

Using a natural language sentence to describe the content of an image is...
research
11/21/2020

Deep learning for video game genre classification

Video game genre classification based on its cover and textual descripti...
research
04/09/2022

On the Importance of Karaka Framework in Multi-modal Grounding

Computational Paninian Grammar model helps in decoding a natural languag...
research
02/13/2023

Large Scale Multi-Lingual Multi-Modal Summarization Dataset

Significant developments in techniques such as encoder-decoder models ha...
research
05/28/2019

Miss Tools and Mr Fruit: Emergent communication in agents learning about object affordances

Recent research studies communication emergence in communities of deep n...

Please sign up or login with your details

Forgot password? Click here to reset