Log In Sign Up

Assessing Transferability from Simulation to Reality for Reinforcement Learning

by   Fabio Muratore, et al.

Learning robot control policies from physics simulations is of great interest to the robotics community as it may render the learning process faster, cheaper, and safer by alleviating the need for expensive real-world experiments. However, the direct transfer of learned behavior from simulation to reality is a major challenge. Optimizing a policy on a slightly faulty simulator can easily lead to the maximization of the 'Simulation Optimization Bias' (SOB). In this case, the optimizer exploits modeling errors of the simulator such that the resulting behavior can potentially damage the robot. We tackle this challenge by applying domain randomization, i.e., randomizing the parameters of the physics simulations during learning. We propose an algorithm called Simulation-based Policy Optimization with Transferability Assessment (SPOTA) which uses an estimator of the SOB to formulate a stopping criterion for training. The introduced estimator quantifies the over-fitting to the set of domains experienced while training. Our experimental results in two different environments show that the new simulation-based policy search algorithm is able to learn a control policy exclusively from a randomized simulator, which can be applied directly to real system without any additional training on the latter.


page 8

page 12

page 14


Bayesian Domain Randomization for Sim-to-Real Transfer

When learning policies for robot control, the real-world data required i...

Distilled Domain Randomization

Deep reinforcement learning is an effective tool to learn robot control ...

Robot Learning from Randomized Simulations: A Review

The rise of deep learning has caused a paradigm shift in robotics resear...

BayesSim: adaptive domain randomization via probabilistic inference for robotics simulators

We introduce BayesSim, a framework for robotics simulations allowing a f...

Policy Transfer via Kinematic Domain Randomization and Adaptation

Transferring reinforcement learning policies trained in physics simulati...

Aggressive Quadrotor Flight Using Curiosity-Driven Reinforcement Learning

The ability to perform aggressive movements, which are called aggressive...

Sim-to-real reinforcement learning applied to end-to-end vehicle control

In this work, we study vision-based end-to-end reinforcement learning on...