Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping

by   Konstantinos Bousmalis, et al.

Instrumenting and collecting annotated visual grasping datasets to train modern machine learning algorithms can be extremely time-consuming and expensive. An appealing alternative is to use off-the-shelf simulators to render synthetic data for which ground-truth annotations are generated automatically. Unfortunately, models trained purely on simulated data often fail to generalize to the real world. We study how randomized simulated environments and domain adaptation methods can be extended to train a grasping system to grasp novel objects from raw monocular RGB images. We extensively evaluate our approaches with a total of more than 25,000 physical test grasps, studying a range of simulation conditions and domain adaptation methods, including a novel extension of pixel-level domain adaptation that we term the GraspGAN. We show that, by using synthetic data and domain adaptation, we are able to reduce the number of real-world samples needed to achieve a given level of performance by up to 50 times, using only randomly generated simulated objects. We also show that by using only unlabeled real-world data and our GraspGAN methodology, we obtain real-world grasping performance without any real-world labels that is similar to that achieved with 939,777 labeled real-world samples.


page 1

page 3

page 5

page 6


Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks

Collecting well-annotated image datasets to train modern machine learnin...

Bi-directional Domain Adaptation for Sim2Real Transfer of Embodied Navigation Agents

Deep reinforcement learning models are notoriously data hungry, yet real...

Play and Learn: Using Video Games to Train Computer Vision Models

Video games are a compelling source of annotated data as they can readil...

Multi-task Domain Adaptation for Deep Learning of Instance Grasping from Simulation

Learning-based approaches to robotic manipulation are limited by the sca...

VoloGAN: Adversarial Domain Adaptation for Synthetic Depth Data

We present VoloGAN, an adversarial domain adaptation network that transl...

Automatic Model Based Dataset Generation for Fast and Accurate Crop and Weeds Detection

Selective weeding is one of the key challenges in the field of agricultu...

Fast robust peg-in-hole insertion with continuous visual servoing

This paper demonstrates a visual servoing method which is robust towards...

Please sign up or login with your details

Forgot password? Click here to reset