Bias-Reduced Hindsight Experience Replay with Virtual Goal Prioritization

05/14/2019
by   Binyamin Manela, et al.
0

Hindsight Experience Replay (HER) is a multi-goal reinforcement learning algorithm for sparse reward functions. The algorithm treats every failure as a success for an alternative (virtual) goal that has been achieved in the episode. Virtual goals are randomly selected, irrespective of which are most instructive for the agent. In this paper, we present two improvements over the existing HER algorithm. First, we prioritize virtual goals from which the agent will learn more valuable information. We call this property the instructiveness of the virtual goal and define it by a heuristic measure, which expresses how well the agent will be able to generalize from that virtual goal to actual goals. Secondly, we reduce existing bias in HER by the removal of misleading samples. To test our algorithms, we built two challenging environments with sparse reward functions. Our empirical results in both environments show vast improvement in the final success rate and sample efficiency when compared to the original HER algorithm.

READ FULL TEXT
research
01/12/2020

Deep Reinforcement Learning for Complex Manipulation Tasks with Sparse Feedback

Learning optimal policies from sparse feedback is a known challenge in r...
research
05/21/2019

Maximum Entropy-Regularized Multi-Goal Reinforcement Learning

In Multi-Goal Reinforcement Learning, an agent learns to achieve multipl...
research
09/16/2018

Improvements on Hindsight Learning

Sparse reward problems are one of the biggest challenges in Reinforcemen...
research
02/12/2019

ACTRCE: Augmenting Experience via Teacher's Advice For Multi-Goal Reinforcement Learning

Sparse reward is one of the most challenging problems in reinforcement l...
research
08/31/2022

Cluster-based Sampling in Hindsight Experience Replay for Robot Control

In multi-goal reinforcement learning in an environment, agents learn pol...
research
10/02/2018

Energy-Based Hindsight Experience Prioritization

In Hindsight Experience Replay (HER), a reinforcement learning agent is ...
research
08/28/2020

Sample Efficiency in Sparse Reinforcement Learning: Or Your Money Back

Sparse rewards present a difficult problem in reinforcement learning and...

Please sign up or login with your details

Forgot password? Click here to reset