Emergent Communication with World Models

We introduce Language World Models, a class of language-conditional generative model which interpret natural language messages by predicting latent codes of future observations. This provides a visual grounding of the message, similar to an enhanced observation of the world, which may include objects outside of the listening agent's field-of-view. We incorporate this "observation" into a persistent memory state, and allow the listening agent's policy to condition on it, akin to the relationship between memory and controller in a World Model. We show this improves effective communication and task success in 2D gridworld speaker-listener navigation tasks. In addition, we develop two losses framed specifically for our model-based formulation to promote positive signalling and positive listening. Finally, because messages are interpreted in a generative model, we can visualize the model beliefs to gain insight into how the communication channel is utilized.

READ FULL TEXT

page 2

page 8

page 11

research
04/23/2017

Translating Neuralese

Several approaches have recently been proposed for learning decentralize...
research
06/17/2019

Visual Navigation by Generating Next Expected Observations

We propose a novel approach to visual navigation in unknown environments...
research
08/17/2023

Language-enhanced RNR-Map: Querying Renderable Neural Radiance Field maps with natural language

We present Le-RNR-Map, a Language-enhanced Renderable Neural Radiance ma...
research
02/28/2023

On the Role of Emergent Communication for Social Learning in Multi-Agent Reinforcement Learning

Explicit communication among humans is key to coordinating and learning....
research
10/14/2019

Dynamic Attention Networks for Task Oriented Grounding

In order to successfully perform tasks specified by natural language ins...
research
03/02/2012

Modelling Social Structures and Hierarchies in Language Evolution

Language evolution might have preferred certain prior social configurati...
research
08/06/2020

Assisted Perception: Optimizing Observations to Communicate State

We aim to help users estimate the state of the world in tasks like robot...

Please sign up or login with your details

Forgot password? Click here to reset