Policy Transfer via Kinematic Domain Randomization and Adaptation

by   Ioannis Exarchos, et al.

Transferring reinforcement learning policies trained in physics simulation to the real hardware remains a challenge, known as the "sim-to-real" gap. Domain randomization is a simple yet effective technique to address dynamics discrepancies across source and target domains, but its success generally depends on heuristics and trial-and-error. In this work we investigate the impact of randomized parameter selection on policy transferability across different types of domain discrepancies. Contrary to common practice in which kinematic parameters are carefully measured while dynamic parameters are randomized, we found that virtually randomizing kinematic parameters (e.g., link lengths) during training in simulation generally outperforms dynamic randomization. Based on this finding, we introduce a new domain adaptation algorithm that utilizes simulated kinematic parameters variation. Our algorithm, Multi-Policy Bayesian Optimization, trains an ensemble of universal policies conditioned on virtual kinematic parameters and efficiently adapts to the target environment using a limited number of target domain rollouts. We showcase our findings on a simulated quadruped robot in five different target environments covering different aspects of domain discrepancies.


page 1

page 5

page 6


Bayesian Domain Randomization for Sim-to-Real Transfer

When learning policies for robot control, the real-world data required i...

Active Domain Randomization

Domain randomization is a popular technique for improving domain transfe...

Policy Transfer with Strategy Optimization

Computer simulation provides an automatic and safe way for training robo...

Distilled Domain Randomization

Deep reinforcement learning is an effective tool to learn robot control ...

Robust Domain Randomization for Reinforcement Learning

Producing agents that can generalize to a wide range of environments is ...

Assessing Transferability from Simulation to Reality for Reinforcement Learning

Learning robot control policies from physics simulations is of great int...

Not Only Domain Randomization: Universal Policy with Embedding System Identification

Domain randomization (DR) cannot provide optimal policies for adapting t...