DANLI: Deliberative Agent for Following Natural Language Instructions

10/22/2022
by   Yichi Zhang, et al.
0

Recent years have seen an increasing amount of work on embodied AI agents that can perform tasks by following human language instructions. However, most of these agents are reactive, meaning that they simply learn and imitate behaviors encountered in the training data. These reactive agents are insufficient for long-horizon complex tasks. To address this limitation, we propose a neuro-symbolic deliberative agent that, while following language instructions, proactively applies reasoning and planning based on its neural and symbolic representations acquired from past experience (e.g., natural language and egocentric vision). We show that our deliberative agent achieves greater than 70 benchmark. Moreover, the underlying reasoning and planning processes, together with our modular framework, offer impressive transparency and explainability to the behaviors of the agent. This enables an in-depth understanding of the agent's capabilities, which shed light on challenges and opportunities for future embodied agents for instruction following. The code is available at https://github.com/sled-group/DANLI.

READ FULL TEXT

page 3

page 14

page 17

page 18

research
12/06/2020

MOCA: A Modular Object-Centric Approach for Interactive Instruction Following

Performing simple household tasks based on language directives is very n...
research
01/02/2022

The Introspective Agent: Interdependence of Strategy, Physiology, and Sensing for Embodied Agents

The last few years have witnessed substantial progress in the field of e...
research
06/21/2023

Improving Long-Horizon Imitation Through Instruction Prediction

Complex, long-horizon planning and its combinatorial nature pose steep c...
research
10/13/2021

Improving the Robustness to Variations of Objects and Instructions with a Neuro-Symbolic Approach for Interactive Instruction Following

An interactive instruction following task has been proposed as a benchma...
research
11/07/2022

Prompter: Utilizing Large Language Model Prompting for a Data Efficient Embodied Instruction Following

Embodied Instruction Following (EIF) studies how mobile manipulator robo...
research
09/16/2021

Hierarchical Control of Situated Agents through Natural Language

When humans conceive how to perform a particular task, they do so hierar...
research
11/21/2019

Teaching Perception

The visual world is very rich and generally too complex to perceive in i...

Please sign up or login with your details

Forgot password? Click here to reset