FollowNet: Robot Navigation by Following Natural Language Directions with Deep Reinforcement Learning

05/16/2018
by   Pararth Shah, et al.
0

Understanding and following directions provided by humans can enable robots to navigate effectively in unknown situations. We present FollowNet, an end-to-end differentiable neural architecture for learning multi-modal navigation policies. FollowNet maps natural language instructions as well as visual and depth inputs to locomotion primitives. FollowNet processes instructions using an attention mechanism conditioned on its visual and depth input to focus on the relevant parts of the command while performing the navigation task. Deep reinforcement learning (RL) a sparse reward learns simultaneously the state representation, the attention function, and control policies. We evaluate our agent on a dataset of complex natural language directions that guide the agent through a rich and realistic dataset of simulated homes. We show that the FollowNet agent learns to execute previously unseen instructions described with a similar vocabulary, and successfully navigates along paths not encountered during training. The agent shows 30 improvement over a baseline model without the attention mechanism, with 52 success rate at novel instructions.

READ FULL TEXT

page 1

page 3

page 4

page 5

page 6

research
10/14/2022

AVLEN: Audio-Visual-Language Embodied Navigation in 3D Environments

Recent years have seen embodied visual navigation advance in two distinc...
research
06/01/2020

Translating Natural Language Instructions for Behavioral Robot Navigation with a Multi-Head Attention Mechanism

We propose a multi-head attention mechanism as a blending layer in a neu...
research
12/21/2018

Learning to Navigate the Web

Learning in environments with large state and action spaces, and sparse ...
research
09/19/2019

RUN through the Streets: A New Dataset and Baseline Models for Realistic Urban Navigation

Following navigation instructions in natural language requires a composi...
research
02/18/2023

Natural Language-conditioned Reinforcement Learning with Inside-out Task Language Development and Translation

Natural Language-conditioned reinforcement learning (RL) enables the age...
research
10/09/2019

Fast Task-Adaptation for Tasks Labeled Using Natural Language in Reinforcement Learning

Over its lifetime, a reinforcement learning agent is often tasked with d...
research
07/21/2020

Soft Expert Reward Learning for Vision-and-Language Navigation

Vision-and-Language Navigation (VLN) requires an agent to find a specifi...

Please sign up or login with your details

Forgot password? Click here to reset