DeepAI AI Chat
Log In Sign Up

Improvisation through Physical Understanding: Using Novel Objects as Tools with Visual Foresight

by   Annie Xie, et al.

Machine learning techniques have enabled robots to learn narrow, yet complex tasks and also perform broad, yet simple skills with a wide variety of objects. However, learning a model that can both perform complex tasks and generalize to previously unseen objects and goals remains a significant challenge. We study this challenge in the context of "improvisational" tool use: a robot is presented with novel objects and a user-specified goal (e.g., sweep some clutter into the dustpan), and must figure out, using only raw image observations, how to accomplish the goal using the available objects as tools. We approach this problem by training a model with both a visual and physical understanding of multi-object interactions, and develop a sampling-based optimizer that can leverage these interactions to accomplish tasks. We do so by combining diverse demonstration data with self-supervised interaction data, aiming to leverage the interaction data to build generalizable models and the demonstration data to guide the model-based RL planner to solve complex tasks. Our experiments show that our approach can solve a variety of complex tool use tasks from raw pixel inputs, outperforming both imitation learning and self-supervised learning individually. Furthermore, we show that the robot can perceive and use novel objects as tools, including objects that are not conventional tools, while also choosing dynamically to use or not use tools depending on whether or not they are required.


page 1

page 2

page 3

page 4

page 5

page 6

page 7

page 8


Contextual Imagined Goals for Self-Supervised Robotic Learning

While reinforcement learning provides an appealing formalism for learnin...

GIFT: Generalizable Interaction-aware Functional Tool Affordances without Labels

Tool use requires reasoning about the fit between an object's affordance...

Learning at the Ends: From Hand to Tool Affordances in Humanoid Robots

One of the open challenges in designing robots that operate successfully...

Robustness via Retrying: Closed-Loop Robotic Manipulation with Self-Supervised Learning

Prediction is an appealing objective for self-supervised learning of beh...

ToolTango: Common sense Generalization in Predicting Sequential Tool Interactions for Robot Plan Synthesis

Robots assisting us in environments such as factories or homes must lear...

An open-ended learning architecture to face the REAL 2020 simulated robot competition

Open-ended learning is a core research field of machine learning and rob...

ToolNet: Using Commonsense Generalization for Predicting Tool Use for Robot Plan Synthesis

A robot working in a physical environment (like home or factory) needs t...