Synthetically Trained Neural Networks for Learning Human-Readable Plans from Real-World Demonstrations

05/18/2018
by   Jonathan Tremblay, et al.
0

We present a system to infer and execute a human-readable program from a real-world demonstration. The system consists of a series of neural networks to perform perception, program generation, and program execution. Leveraging convolutional pose machines, the perception network reliably detects the bounding cuboids of objects in real images even when severely occluded, after training only on synthetic images using domain randomization. To increase the applicability of the perception network to new scenarios, the network is formulated to predict in image space rather than in world space. Additional networks detect relationships between objects, generate plans, and determine actions to reproduce a real-world demonstration. The networks are trained entirely in simulation, and the system is tested in the real world on the pick-and-place problem of stacking colored cubes using a Baxter robot.

READ FULL TEXT

page 1

page 4

page 5

page 8

research
03/10/2019

Domain Randomization for Active Pose Estimation

Accurate state estimation is a fundamental component of robotic control....
research
11/04/2019

Learning One-Shot Imitation from Humans without Humans

Humans can naturally learn to execute a new task by seeing it performed ...
research
07/04/2018

Transfer Learning From Synthetic To Real Images Using Variational Autoencoders For Precise Position Detection

Capturing and labeling camera images in the real world is an expensive t...
research
11/17/2020

Sim-to-Real Task Planning and Execution from Perception via Reactivity and Recovery

Zero-shot execution of unseen robotic tasks is an important problem in r...
research
05/10/2021

Towards Robust One-shot Task Execution using Knowledge Graph Embeddings

Requiring multiple demonstrations of a task plan presents a burden to en...
research
02/01/2022

ADG-Pose: Automated Dataset Generation for Real-World Human Pose Estimation

Recent advancements in computer vision have seen a rise in the prominenc...
research
04/13/2019

Recovery of Superquadrics from Range Images using Deep Learning: A Preliminary Study

It has been a longstanding goal in computer vision to describe the 3D ph...

Please sign up or login with your details

Forgot password? Click here to reset