FILM: Following Instructions in Language with Modular Methods

10/12/2021
by   So Yeon Min, et al.
1

Recent methods for embodied instruction following are typically trained end-to-end using imitation learning. This requires the use of expert trajectories and low-level language instructions. Such approaches assume learned hidden states will simultaneously integrate semantics from the language and vision to perform state tracking, spatial memory, exploration, and long-term planning. In contrast, we propose a modular method with structured representations that (1) builds a semantic map of the scene, and (2) performs exploration with a semantic search policy, to achieve the natural language goal. Our modular method achieves SOTA performance (24.46 (8.17 both expert trajectories and low-level instructions. Leveraging low-level language, however, can further increase our performance (26.49 suggest that an explicit spatial memory and a semantic search policy can provide a stronger and more general representation for state-tracking and guidance, even in the absence of expert trajectories or low-level instructions.

READ FULL TEXT

page 2

page 4

page 6

page 9

page 13

research
05/31/2018

Following High-level Navigation Instructions on a Simulated Quadcopter with Imitation Learning

We introduce a method for following high-level navigation instructions b...
research
03/10/2021

ELLA: Exploration through Learned Language Abstraction

Building agents capable of understanding language instructions is critic...
research
02/25/2022

SGL: Symbolic Goal Learning in a Hybrid, Modular Framework for Human Instruction Following

This paper investigates robot manipulation based on human instruction wi...
research
01/26/2022

Learning Invariable Semantical Representation from Language for Extensible Policy Generalization

Recently, incorporating natural language instructions into reinforcement...
research
07/12/2021

A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution

Natural language provides an accessible and expressive interface to spec...
research
07/27/2023

Imitating Complex Trajectories: Bridging Low-Level Stability and High-Level Behavior

We propose a theoretical framework for studying the imitation of stochas...
research
11/18/2016

Fast low-level pattern matching algorithm

This paper focuses on pattern matching in the DNA sequence. It was inspi...

Please sign up or login with your details

Forgot password? Click here to reset