Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards

by   Matej Večerík, et al.

We propose a general and model-free approach for Reinforcement Learning (RL) on real robotics with sparse rewards. We build upon the Deep Deterministic Policy Gradient (DDPG) algorithm to use demonstrations. Both demonstrations and actual interactions are used to fill a replay buffer and the sampling ratio between demonstrations and transitions is automatically tuned via a prioritized replay mechanism. Typically, carefully engineered shaping rewards are required to enable the agents to efficiently explore on high dimensional control problems such as robotics. They are also required for model-based acceleration methods relying on local solvers such as iLQG (e.g. Guided Policy Search and Normalized Advantage Function). The demonstrations replace the need for carefully engineered rewards, and reduce the exploration problem encountered by classical RL approaches in these domains. Demonstrations are collected by a robot kinesthetically force-controlled by a human demonstrator. Results on four simulated insertion tasks show that DDPG from demonstrations out-performs DDPG, and does not require engineered rewards. Finally, we demonstrate the method on a real robotics task consisting of inserting a clip (flexible object) into a rigid object.


page 4

page 5

page 6


Overcoming Exploration in Reinforcement Learning with Demonstrations

Exploration in environments with sparse rewards has been a persistent pr...

Learning Sparse Control Tasks from Pixels by Latent Nearest-Neighbor-Guided Explorations

Recent progress in deep reinforcement learning (RL) and computer vision ...

Reinforcement Learning for Robotic Manipulation using Simulated Locomotion Demonstrations

Learning robot manipulation policies through reinforcement learning (RL)...

Align-RUDDER: Learning From Few Demonstrations by Reward Redistribution

Reinforcement Learning algorithms require a large number of samples to s...

Extending Deep Model Predictive Control with Safety Augmented Value Estimation from Demonstrations

Reinforcement learning (RL) for robotics is challenging due to the diffi...

Gaussian Processes for Data-Efficient Learning in Robotics and Control

Autonomous learning has been a promising direction in control and roboti...

Dealing with Sparse Rewards in Continuous Control Robotics via Heavy-Tailed Policies

In this paper, we present a novel Heavy-Tailed Stochastic Policy Gradien...

Please sign up or login with your details

Forgot password? Click here to reset