Language Understanding for Field and Service Robots in a Priori Unknown Environments

05/21/2021
by   Matthew R. Walter, et al.
6

Contemporary approaches to perception, planning, estimation, and control have allowed robots to operate robustly as our remote surrogates in uncertain, unstructured environments. There is now an opportunity for robots to operate not only in isolation, but also with and alongside humans in our complex environments. Natural language provides an efficient and flexible medium through which humans can communicate with collaborative robots. Through significant progress in statistical methods for natural language understanding, robots are now able to interpret a diverse array of free-form navigation, manipulation, and mobile manipulation commands. However, most contemporary approaches require a detailed prior spatial-semantic map of the robot's environment that models the space of possible referents of the utterance. Consequently, these methods fail when robots are deployed in new, previously unknown, or partially observed environments, particularly when mental models of the environment differ between the human operator and the robot. This paper provides a comprehensive description of a novel learning framework that allows field and service robots to interpret and correctly execute natural language instructions in a priori unknown, unstructured environments. Integral to our approach is its use of language as a "sensor" – inferring spatial, topological, and semantic information implicit in natural language utterances and then exploiting this information to learn a distribution over a latent environment model. We incorporate this distribution in a probabilistic language grounding model and infer a distribution over a symbolic representation of the robot's action space. We use imitation learning to identify a belief space policy that reasons over the environment and behavior distributions. We evaluate our framework through a variety of different navigation and mobile manipulation experiments.

READ FULL TEXT

page 3

page 19

page 22

page 23

page 26

page 28

page 30

page 31

research
10/22/2019

Language-guided Semantic Mapping and Mobile Manipulation in Partially Observable Environments

Recent advances in data-driven models for grounded language understandin...
research
09/27/2013

Madeup: A Mobile Development Environment for Programming 3-D Models

Constructionism is a learning theory that states that we learn more when...
research
06/15/2023

Tell Me Where to Go: A Composable Framework for Context-Aware Embodied Robot Navigation

Humans have the remarkable ability to navigate through unfamiliar enviro...
research
03/21/2019

Inferring Compact Representations for Efficient Natural Language Understanding of Robot Instructions

The speed and accuracy with which robots are able to interpret natural l...
research
03/12/2020

Natural Language Interaction to Facilitate Mental Models of Remote Robots

Increasingly complex and autonomous robots are being deployed in real-wo...
research
05/16/2023

Accessible Interfaces for the Development and Deployment of Robotic Platforms

Accessibility is one of the most important features in the design of rob...
research
12/04/2020

Spatial Language Understanding for Object Search in Partially Observed Cityscale Environments

We present a system that enables robots to interpret spatial language as...

Please sign up or login with your details

Forgot password? Click here to reset