Imaginary Hindsight Experience Replay: Curious Model-based Learning for Sparse Reward Tasks

10/05/2021
by   Robert McCarthy, et al.
0

Model-based reinforcement learning is a promising learning strategy for practical robotic applications due to its improved data-efficiency versus model-free counterparts. However, current state-of-the-art model-based methods rely on shaped reward signals, which can be difficult to design and implement. To remedy this, we propose a simple model-based method tailored for sparse-reward multi-goal tasks that foregoes the need for complicated reward engineering. This approach, termed Imaginary Hindsight Experience Replay, minimises real-world interactions by incorporating imaginary data into policy updates. To improve exploration in the sparse-reward setting, the policy is trained with standard Hindsight Experience Replay and endowed with curiosity-based intrinsic rewards. Upon evaluation, this approach provides an order of magnitude increase in data-efficiency on average versus the state-of-the-art model-free method in the benchmark OpenAI Gym Fetch Robotics tasks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/01/2020

PlanGAN: Model-based Planning With Sparse Rewards and Multiple Goals

Learning with sparse rewards remains a significant challenge in reinforc...
research
06/28/2023

RoMo-HER: Robust Model-based Hindsight Experience Replay

Sparse rewards are one of the factors leading to low sample efficiency i...
research
06/12/2019

When to use parametric models in reinforcement learning?

We examine the question of when and how parametric models are most usefu...
research
06/25/2018

Multi-objective Model-based Policy Search for Data-efficient Learning with Sparse Rewards

The most data-efficient algorithms for reinforcement learning in robotic...
research
11/04/2019

Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewards

While using shaped rewards can be beneficial when solving sparse reward ...
research
02/07/2021

Model-Augmented Q-learning

In recent years, Q-learning has become indispensable for model-free rein...
research
02/25/2021

Bias-reduced multi-step hindsight experience replay

Multi-goal reinforcement learning is widely used in planning and robot m...

Please sign up or login with your details

Forgot password? Click here to reset