Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks

by   Stephen James, et al.

Real world data, especially in the domain of robotics, is notoriously costly to collect. One way to circumvent this can be to leverage the power of simulation in order to produce large amounts of labelled data. However, training models on simulated images does not readily transfer to real-world ones. Using domain adaptation methods to cross this "reality gap" requires at best a large amount of unlabelled real-world data, whilst domain randomization alone can waste modeling power, rendering certain reinforcement learning (RL) methods unable to learn the task of interest. In this paper, we present Randomized-to-Canonical Adaptation Networks (RCANs), a novel approach to crossing the visual reality gap that uses no real-world data. Our method learns to translate randomized rendered images into their equivalent non-randomized, canonical versions. This in turn allows for real images to also be translated into canonical sim images. We demonstrate the effectiveness of this sim-to-real approach by training a vision-based closed-loop grasping reinforcement learning agent in simulation, and then transferring it to the real world to attain 70 zero-shot grasp success on unseen objects, a result that almost doubles the success of learning the same task directly on domain randomization alone. Additionally, by joint finetuning in the real-world with only 5,000 real-world grasps, our method achieves 91 trained with 580,000 real-world grasps, resulting in a reduction of real-world data by more than 99


page 1

page 4

page 5

page 7

page 14


A general approach to bridge the reality-gap

Employing machine learning models in the real world requires collecting ...

Randomized-to-Canonical Model Predictive Control for Real-world Visual Robotic Manipulation

Many works have recently explored Sim-to-real transferable visual model ...

RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Real

Deep neural network based reinforcement learning (RL) can learn appropri...

DR2L: Surfacing Corner Cases to Robustify Autonomous Driving via Domain Randomization Reinforcement Learning

How to explore corner cases as efficiently and thoroughly as possible ha...

Unknown Object Segmentation through Domain Adaptation

The ability to segment unknown objects in cluttered scenes has a profoun...

Closing the Sim2Real Gap in Dynamic Cloth Manipulation

Cloth manipulation is a challenging task due to the many degrees of free...

Zero-Shot Reinforcement Learning with Deep Attention Convolutional Neural Networks

Simulation-to-simulation and simulation-to-real world transfer of neural...

Please sign up or login with your details

Forgot password? Click here to reset