Mapping Instructions and Visual Observations to Actions with Reinforcement Learning

04/28/2017
by   Dipendra Misra, et al.
0

We propose to directly map raw visual observations and text input to actions for instruction execution. While existing approaches assume access to structured environment representations or use a pipeline of separately trained models, we learn a single model to jointly reason about linguistic and visual input. We use reinforcement learning in a contextual bandit setting to train a neural network agent. To guide the agent's exploration, we use reward shaping with different forms of supervision. Our approach does not require intermediate representations, planning procedures, or training different models. We evaluate in a simulated environment, and show significant improvements over supervised learning and common reinforcement learning variants.

READ FULL TEXT
research
10/21/2019

Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flight

We propose a joint simulation and real-world learning framework for mapp...
research
01/21/2020

Unsupervisedly Learned Representations: Should the Quest be Over?

There exists a Classification accuracy gap of about 20 methods of genera...
research
02/07/2020

Causally Correct Partial Models for Reinforcement Learning

In reinforcement learning, we can learn a model of future observations a...
research
05/25/2018

Situated Mapping of Sequential Instructions to Actions with Single-step Reward Observation

We propose a learning approach for mapping context-dependent sequential ...
research
01/25/2020

Following Instructions by Imagining and Reaching Visual Goals

While traditional methods for instruction-following typically assume pri...
research
06/14/2022

Visual Radial Basis Q-Network

While reinforcement learning (RL) from raw images has been largely inves...
research
04/06/2019

Reinforcement Learning with Attention that Works: A Self-Supervised Approach

Attention models have had a significant positive impact on deep learning...

Please sign up or login with your details

Forgot password? Click here to reset