A Tale of Two DRAGGNs: A Hybrid Approach for Interpreting Action-Oriented and Goal-Oriented Instructions

07/26/2017
by   Siddharth Karamcheti, et al.
0

Robots operating alongside humans in diverse, stochastic environments must be able to accurately interpret natural language commands. These instructions often fall into one of two categories: those that specify a goal condition or target state, and those that specify explicit actions, or how to perform a given task. Recent approaches have used reward functions as a semantic representation of goal-based commands, which allows for the use of a state-of-the-art planner to find a policy for the given task. However, these reward functions cannot be directly used to represent action-oriented commands. We introduce a new hybrid approach, the Deep Recurrent Action-Goal Grounding Network (DRAGGN), for task grounding and execution that handles natural language from either category as input, and generalizes to unseen environments. Our robot-simulation results demonstrate that a system successfully interpreting both goal-oriented and action-oriented task specifications brings us closer to robust natural language understanding for human-robot interaction.

READ FULL TEXT
research
04/30/2019

Learning from Implicit Information in Natural Language Instructions for Robotic Manipulations

Human-robot interaction often occurs in the form of instructions given f...
research
04/21/2017

Accurately and Efficiently Interpreting Human-Robot Instructions of Varying Granularities

Humans can ground natural language commands to tasks at both abstract an...
research
09/02/2021

Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation

We study the problem of learning a range of vision-based manipulation ta...
research
03/21/2019

Inferring Compact Representations for Efficient Natural Language Understanding of Robot Instructions

The speed and accuracy with which robots are able to interpret natural l...
research
04/11/2022

Correcting Robot Plans with Natural Language Feedback

When humans design cost or goal specifications for robots, they often pr...
research
06/27/2021

Draw Me a Flower: Grounding Formal Abstract Structures Stated in Informal Natural Language

Forming and interpreting abstraction is a core process in human communic...
research
07/20/2021

Neural Abstructions: Abstractions that Support Construction for Grounded Language Learning

Although virtual agents are increasingly situated in environments where ...

Please sign up or login with your details

Forgot password? Click here to reset