Grounded Language Learning in a Simulated 3D World

by   Karl Moritz Hermann, et al.

We are increasingly surrounded by artificially intelligent technology that takes decisions and executes actions on our behalf. This creates a pressing need for general means to communicate with, instruct and guide artificial agents, with human language the most compelling means for such communication. To achieve this in a scalable fashion, agents must be able to relate language to the world and to actions; that is, their understanding of language must be grounded and embodied. However, learning grounded language is a notoriously challenging problem in artificial intelligence research. Here we present an agent that learns to interpret language in a simulated 3D environment where it is rewarded for the successful execution of written instructions. Trained via a combination of reinforcement and unsupervised learning, and beginning with minimal prior knowledge, the agent learns to relate linguistic symbols to emergent perceptual representations of its physical surroundings and to pertinent sequences of actions. The agent's comprehension of language extends beyond its prior experience, enabling it to apply familiar language to unfamiliar situations and to interpret entirely novel instructions. Moreover, the speed with which this agent learns new words increases as its semantic knowledge grows. This facility for generalising and bootstrapping semantic knowledge indicates the potential of the present approach for reconciling ambiguous natural language with the complexity of the physical world.


page 5

page 8

page 9

page 14

page 15


Understanding Grounded Language Learning Agents

Neural network-based systems can now learn to locate the referents of wo...

Learning to Model the World with Language

To interact with humans in the world, agents need to understand the dive...

Grounding Hindsight Instructions in Multi-Goal Reinforcement Learning for Robotics

This paper focuses on robotic reinforcement learning with sparse rewards...

Intensional Artificial Intelligence: From Symbol Emergence to Explainable and Empathetic AI

We argue that an explainable artificial intelligence must possess a rati...

Mastering emergent language: learning to guide in simulated navigation

To cooperate with humans effectively, virtual agents need to be able to ...

Self-Educated Language Agent With Hindsight Experience Replay For Instruction Following

Language creates a compact representation of the world and allows the de...

Towards an Indexical Model of Situated Language Comprehension for Cognitive Agents in Physical Worlds

We propose a computational model of situated language comprehension base...

Please sign up or login with your details

Forgot password? Click here to reset