Sim-to-Real Task Planning and Execution from Perception via Reactivity and Recovery

by   Shohin Mukherjee, et al.

Zero-shot execution of unseen robotic tasks is an important problem in robotics. One potential approach is through task planning: combining known skills based on their preconditions and effects to achieve a user-specified goal. In this work, we propose such a task planning approach to build a reactive system for multi-step manipulation tasks that can be trained on simulation data and applied in the real-world. We explore a block-stacking task because it has a clear structure, where multiple skills must be chained together: pick up a block, place it on top of another block, etc. We learn these skills, along with a set of predicate preconditions and termination conditions, entirely in simulation. All components are learned as PointNet++ models, parameterized by the masks of relevant objects. The predicates allow us to create high-level plans combining different skills. They also serve as precondition functions for the skills, which enables the system to recognize failures and accomplish long-horizon tasks from perceptual input, which is critical for real-world execution. We evaluate our proposed approach in both simulation and in the real-world, showing an increase in success rate from 91.6 as compared with naive baselines.


page 1

page 2

page 3

page 4

page 6


Search-Based Task Planning with Learned Skill Effect Models for Lifelong Robotic Manipulation

Lifelong-learning robots need to be able to acquire new skills and plan ...

Learning Hierarchical Interactive Multi-Object Search for Mobile Manipulation

Existing object-search approaches enable robots to search through free p...

LAAIR: A Layered Architecture for Autonomous Interactive Robots

When developing general purpose robots, the overarching software archite...

Efficient Bimanual Manipulation Using Learned Task Schemas

We address the problem of effectively composing skills to solve sparse-r...

Synthetically Trained Neural Networks for Learning Human-Readable Plans from Real-World Demonstrations

We present a system to infer and execute a human-readable program from a...

Efficiently Learning Recoveries from Failures Under Partial Observability

Operating under real world conditions is challenging due to the possibil...

Discover Life Skills for Planning with Bandits via Observing and Learning How the World Works

We propose a novel approach for planning agents to compose abstract skil...

Please sign up or login with your details

Forgot password? Click here to reset