Summarizing a virtual robot's past actions in natural language

03/13/2022
by   Chad DeChant, et al.
0

We propose and demonstrate the task of giving natural language summaries of the actions of a robotic agent in a virtual environment. We explain why such a task is important, what makes it difficult, and discuss how it might be addressed. To encourage others to work on this, we show how a popular existing dataset that matches robot actions with natural language descriptions designed for an instruction following task can be repurposed to serve as a training ground for robot action summarization work. We propose and test several methods of learning to generate such summaries, starting from either egocentric video frames of the robot taking actions or intermediate text representations of the actions used by an automatic planner. We provide quantitative and qualitative evaluations of our results, which can serve as a baseline for future work.

READ FULL TEXT
research
06/16/2023

Learning to Summarize and Answer Questions about a Virtual Robot's Past Actions

When robots perform long action sequences, users will want to easily and...
research
03/23/2020

Caption Generation of Robot Behaviors based on Unsupervised Learning of Action Segments

Bridging robot action sequences and their natural language captions is a...
research
07/23/2016

Processing Natural Language About Ongoing Actions

Actions may not proceed as planned; they may be interrupted, resumed or ...
research
02/15/2023

ANSEL Photobot: A Robot Event Photographer with Semantic Intelligence

Our work examines the way in which large language models can be used for...
research
07/15/2022

Learning Flexible Translation between Robot Actions and Language Descriptions

Handling various robot action-language translation tasks flexibly is an ...
research
05/27/2019

Harry Potter and the Action Prediction Challenge from Natural Language

We explore the challenge of action prediction from textual descriptions ...
research
10/07/2017

Interactive Learning of State Representation through Natural Language Instruction and Explanation

One significant simplification in most previous work on robot learning i...

Please sign up or login with your details

Forgot password? Click here to reset