DeepAI AI Chat
Log In Sign Up

Sim-to-Real Task Planning and Execution from Perception via Reactivity and Recovery

by   Shohin Mukherjee, et al.
Carnegie Mellon University
University of Washington

Zero-shot execution of unseen robotic tasks is an important problem in robotics. One potential approach is through task planning: combining known skills based on their preconditions and effects to achieve a user-specified goal. In this work, we propose such a task planning approach to build a reactive system for multi-step manipulation tasks that can be trained on simulation data and applied in the real-world. We explore a block-stacking task because it has a clear structure, where multiple skills must be chained together: pick up a block, place it on top of another block, etc. We learn these skills, along with a set of predicate preconditions and termination conditions, entirely in simulation. All components are learned as PointNet++ models, parameterized by the masks of relevant objects. The predicates allow us to create high-level plans combining different skills. They also serve as precondition functions for the skills, which enables the system to recognize failures and accomplish long-horizon tasks from perceptual input, which is critical for real-world execution. We evaluate our proposed approach in both simulation and in the real-world, showing an increase in success rate from 91.6 as compared with naive baselines.


page 1

page 2

page 3

page 4

page 6


Search-Based Task Planning with Learned Skill Effect Models for Lifelong Robotic Manipulation

Lifelong-learning robots need to be able to acquire new skills and plan ...

Efficiently Learning Recoveries from Failures Under Partial Observability

Operating under real world conditions is challenging due to the possibil...

LAAIR: A Layered Architecture for Autonomous Interactive Robots

When developing general purpose robots, the overarching software archite...

Efficient Bimanual Manipulation Using Learned Task Schemas

We address the problem of effectively composing skills to solve sparse-r...

Synthetically Trained Neural Networks for Learning Human-Readable Plans from Real-World Demonstrations

We present a system to infer and execute a human-readable program from a...

Discover Life Skills for Planning with Bandits via Observing and Learning How the World Works

We propose a novel approach for planning agents to compose abstract skil...

Efficient Object Manipulation to an Arbitrary Goal Pose: Learning-based Anytime Prioritized Planning

We focus on the task of object manipulation to an arbitrary goal pose, i...