ARCHER: Aggressive Rewards to Counter bias in Hindsight Experience Replay

09/06/2018 ∙ by Sameera Lanka, et al. ∙ NC State University 6

Experience replay is an important technique for addressing sample-inefficiency in deep reinforcement learning (RL), but faces difficulty in learning from binary and sparse rewards due to disproportionately few successful experiences in the replay buffer. Hindsight experience replay (HER) was recently proposed to tackle this difficulty by manipulating unsuccessful transitions, but in doing so, HER introduces a significant bias in the replay buffer experiences and therefore achieves a suboptimal improvement in sample-efficiency. In this paper, we present an analysis on the source of bias in HER, and propose a simple and effective method to counter the bias, to most effectively harness the sample-efficiency provided by HER. Our method, motivated by counter-factual reasoning and called ARCHER, extends HER with a trade-off to make rewards calculated for hindsight experiences numerically greater than real rewards. We validate our algorithm on two continuous control environments from DeepMind Control Suite - Reacher and Finger, which simulate manipulation tasks with a robotic arm - in combination with various reward functions, task complexities and goal sampling strategies. Our experiments consistently demonstrate that countering bias using more aggressive hindsight rewards increases sample efficiency, thus establishing the greater benefit of ARCHER in RL applications with limited computing budget.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 5

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Humans possess the remarkable capacity to learn from experience efficiently and effectively; even from unsuccessful experience as captured by the Chinese saying – “failure is the mother of success”. Consider how parents teach their kids to play simple block stacking games. In the early stages, parents usually provide positive feedback even when kids do not build the correct configuration. By receiving such signals, kids stay encouraged and cultivate an exploratory approach, and over time refine their skill through memory and corrective feedback. Children who learn from failure have been observed to gain faster mastery at tasks than children who are not allowed to fail [Elliott and Dweck1988]. Computationally formulating this approach will greatly help improve robot autonomy towards the ambitious goal of building machines that learn and think like people [Lake et al.2016].

Deep reinforcement learning (RL) stands as a promising method for training robots to autonomously learn dynamic control policies [Kohl and Stone2004, Kober and Peters2011, Levine et al.2016]. The key advantage of deep RL lies in its capacity to learn generalized policy representations in comparison to specialized domain and task specific hand-engineered policies. However, this advantage comes with a caveat that deep RL requires a practically prohibitive amount of training data and computational time to successfully learn policies across high-dimensional, continuous state and continuous action domains, thus hindering its application in robotics. Deep RL methods are commonly used in conjunction with a binary success/failure reward function. Such a reward mechanism allows for the agent to discover optimal policies intended for successful completion of the task, thus eliminating the need for expert reward specification or the danger of inaccurate reward design [Ng, Harada, and Russell1999]. In high-dimensional continuous control, a characteristic property of most practical robots, a binary reward system presents a challenge when the task is complex and the goal states are rarely encountered. In these sparse reward conditions, the agent receives insufficient reinforcement to discriminate between successful and unsuccessful actions, and is consequently unable to learn an effective policy function to make progress at the task. The sparse-reward problem is particularly intensified when prolonged operation of a physical robot to explore and collect data for training is infeasible, dangerous or expensive. Thus, there exists a pressing demand to improve the sample efficiency of deep RL algorithms.

Hindsight Experience Replay (HER) [Andrychowicz et al.2017] is a method which seeks to learn from experiences that resulted in failure in addition to experiences which led to successful completion of the task. If an agent was tasked to achieve goal but ended achieving a different goal , traditional RL methods consider episode as a failure and do not gain any information from this episode. However, HER proceeds as follows - in the first run, the agent stores the transitions and rewards associated with the real episode pertaining to goal . Then the algorithm replays the episode by recomputing the rewards in hindsight, under the context of achieving goal and thereby incorporates information from an unsuccessful episode to train its policy function. For off-policy RL algorithms such as DDPG [Silver et al.2014], NAF [Gu et al.2016b], DQN [Mnih et al.2015], incorporating HER improves sample-efficiency significantly in RL with sparse and binary rewards.

In this paper, we challenge the central assumption underlying the vanilla HER - that the same trajectory of states and actions can be replayed in hindsight, thereby creating real and hindsight experiences which impact the learning process equivalently, through their corresponding rewards. Our idea is inspired by the well-studied phenomenon of hindsight bias [Fischhoff1975] in behavioural psychology, traditionally defined as the tendency to exaggerate the a priori predictability of outcomes after they become known. We argue that the hindsight experiences introduce bias in RL as the likelihood of replayed experience is overestimated. Our hypothesis suggests there should in fact exist a difference in the computation of rewards associated with real, generated experiences and artificially formulated hindsight experiences. We strive to quantitatively establish this difference by evaluating the relative influence of each in learning the optimal policy for a task at hand. We answer the question - Does there in fact exist a hindsight bias in HER and if so, how do we mathematically justify the bias? Upon identifying the potential source of bias, we propose a solution to offset this bias using a weighted trade-off between real rewards and hindsight rewards. We empirically prove that HER works better when hindsight experiences are rewarded more (i.e., more aggressive), especially with sparse and binary rewards. We title our method ARCHER, Aggressive Rewards to Counter bias in HER. In experiments, ARCHER consistently outperforms the sample-efficiency of vanilla HER as validated on two distinct continuous control domains, finger and reacher, tested in simulation using the DeepMind Control Suite [Tassa et al.2018].

2 Related Work

The high sample complexity of deep neural networks compounded with constant policy revision in RL requires that an agent must continually generate vast amount of experiences consistent with its current policy, which then “expire” once the networks have been updated and are therefore discarded after a single use. Sample-inefficiency in deep RL remains a long-standing open challenge and many techniques have been proposed to tackle this issue.

Experience Replay (ER), first introduced in [Lin1992], is a crucial component to stabilize convergence in off-policy deep RL networks, as demonstrated by the successful Deep Q-Network [Mnih et al.2015] algorithm. ER dissects episodes into experience quadruples of the form , where the elements represent current state, action, next state and reward respectively. The experiences are stored in a collective database termed as replay buffer, from where they undergo uniform random sampling to form minibatches for training the RL networks. In addition to allowing for temporal independence in the training set, this approach enables the reuse of past experiences, as any single experience may be sampled multiple times, and hence increases sample-efficiency.

Subsequent research aimed to find more effective sampling strategies and compositions of the replay buffer. Prioritized Experience Replay (PER) [Schaul et al.2015] achieves higher sample efficiency by selecting experiences from the replay buffer according to a frequency distribution prioritizing the importance of each individual transition. [Zhang and Sutton2017] investigates the effect of size of the replay buffer on performance and proposes Combined Experience Replay (CER) to alleviate the liability of a large replay buffer. Hindsight Experience Replay (HER) [Andrychowicz et al.2017], described in greater detail in the following section, strategically augments the replay buffer by reformulating unsuccessful episodes as successful transitions accomplishing a different goal. The resulting balance in successful and unsuccessful experiences in the replay buffer overcomes the disadvantage of sparse rewards.

Hindsight bias was first documented by Fischhoff [Fischhoff1975], referring to the inflation in people’s predicted likelihood of the true outcome of an event, after the outcome is known. This phenomenon, also termed as “creeping determinism”, is one of the most pervasive cognitive biases and routinely affects judgments in multiple domains, including medical diagnoses [Arkes et al.1988], criminal justice [Casper, Benedict, and Perry1989] and financial systems [Anderson, Lowe, and Reckers1993].

Parallel methods to improve sample-efficiency of deep RL include learning hierarchical abstractions [Kulkarni et al.2016, Riedmiller et al.2018]

, reducing variance in policy gradients 

[Gu et al.2016a], model-based algorithms [Deisenroth and Rasmussen2011] and effective exploration [Hester and Stone2013, Pathak et al.2017].

3 ARCHER: Aggressive Rewards to Counter bias in HER

In this section, we provide a brief mathematical introduction to Hindsight Experience Replay (HER) [Andrychowicz et al.2017] using the DDPG algorithm [Silver et al.2014] to be self-contained. Then we present our proposed method, ARCHER, in detail.

3.1 Background on DDPG+HER

Let and denote the state-space and the action-space of the environment respectively, and let represent the space of goals we wish to achieve. We assume that for every state , there exists some goal achieved in that state. This gives rise to the mapping where denotes the achieved goal in state . Moreover, we state that every goal corresponds to the predicate , which indicates whether the goal has been realized in state . The true objective of an agent is to reach a state such that , following which it receives a successful reward signal from the environment.

In environments with complex tasks and sparse rewards, it is extremely unlikely that the agent achieves the goal. Hence, for most of the encountered states, and the agent largely only receives unsuccessful rewards. To solve the deficit of successful experiences, and encourage the agent to effectively discriminate between good and bad policies, we exploit the knowledge that although the agent has not achieved the pre-specified goal , it has achieved goal , and by extension, . We call the hindsight goal and we now explain how to incorporate HER into the off-policy RL algorithm, Deep Deterministic Policy Gradient (DDPG).

DDPG architecture consists of two neural networks - an actor , representing a deterministic policy function parameterized by and a critic which calculates the Q-value of state-action pairs and is parameterized by ; for all and .

To train the actor and critic networks using HER, the state inputs to the networks are appended with the desired goal for the episode. Let

denote the state vector concatenated with the goal. We write the actor and the critic equations as,

(1)
(2)

To stabilize training, we maintain two gradually updated duplicates of the original actor and critic networks, denoted as target actor and target critic . The target networks are not optimized directly by gradient descent and are only used to compute the target Q-values Eqn. (7) for the critic loss Eqn. (6).

Training DDPG using HER consists of two phases. In the first phase, we sample a target goal and an initial state at the start of every episode. Then for every time step , where is the length of the episode, we run the policy network and add the generated real experience tuples, , to the replay buffer.

(3)

where , i.e. adding noise to the actor output to facilitate exploration, and reward is computed by a reward function dependent on the state transitions and goal. For example, if we adopt negative reward for failure and we have,

(4)

This phase is known as standard experience replay. In the second phase, we execute hindsight experience replay where we pretend that the goal we intended to achieve was the goal corresponding to the state of the environment at the terminal step of the episode (other strategies can be used, see [Andrychowicz et al.2017]). We modify the sequence of transitions generated during standard experience replay, by replacing the real goal with the achieved goal . We supplement the replay buffer with hindsight experience tuples,

(5)

where the reward is recomputed at each step in accord with the fake goal .

Thus, the replay buffer is augmented with additional training data containing successful reinforcement signals, mitigating the sparse reward problem. The successful transitions which accompany the HER increase the sample-efficiency of DDPG and help the agent learn policies for high-dimensional continuous control.

The actor and critic are optimized by stochastic gradient descent 

[Rumelhart, Hinton, and Williams1986] by uniformly sampling mini-batches from the replay buffer. Given a mini-batch consisting of experience tuples (each can be either a real experience or hindsight experience), the critic network is updated by minimizing the critic loss between the predicted Q-value and the temporal difference target [Sutton1988],

(6)

where the target value is computed by the target networks,

(7)

and is a predefined discount factor. The actor is trained to maximize the output of the critic by minimizing the following loss using deterministic policy gradient [Silver et al.2014],

(8)

The target networks are updated in a conservative way by,

(9)
(10)

where is a small positive real value.

Given:
       an off-policy RL algorithm , e.g. DDPG
       a strategy for sampling goals for replay
       a reward function
       real reward weight , hindsight reward weight
Initialize
Initialize replay buffer
for episode = 1,  do
     Sample a goal and initial state
     for  do
         Sample an action using the behavior policy from :
               
         Execute the action and observe a new state      
     for  do
         
         Store the transition in
         Sample a set of additional goals for replay,
         for  do
              
              Store the transition in HER with weighted rewards               
     for  do
         Sample a minibatch from the replay buffer
         Perform one step of optimization using and the minibatch      
Algorithm 1 ARCHER

3.2 The Proposed Method: ARCHER

In this section, we first examine the source of bias in HER, and then present our algorithm ARCHER which uses more aggressive rewards for hindsight experiences to combat the bias, and thus achieving greater sample-efficiency.

Compare the real experience tuple in (3) to the artificially constructed hindsight experience tuple in (5). This conversion of the a real experience to its corresponding hindsight experience makes the following unjustified assumption - Given different inputs and , the policy network , returns the same action,

. This assumption overestimates the probability assigned by the policy network to

, given the input . If we actually execute the policy network with as input, it is unlikely to output , making also unlikely. Hence we observe a chain of compounding uncertainty along the hinsight episode.

Therefore, to more effectively use HER, we require to correct the hindsight bias induced by this overestimated probability. The intuitive check would be to generate hindsight experiences by using models capable of counterfactual reasoning, i.e. by asking the network what if was the actual goal, instead of mere substitution of real experiences. However, this a critical limitation of deductive learning models [Pearl2018] and remains a challenge for the future.

We propose a simple solution to offset this bias. We make that case that a hindsight experience and a real experience cannot be treated in the same manner as real experiences are authentically generated by interacting with the environment, and hence their probability is unbiased. In contrast, to overcome hindsight bias, we need to match the true probability of the hindsight experiences to their biased probability. To do so, we nudge the current policy to be more consistent with the hindsight data in the replay buffer. Hence, to meet the overestimated hindsight likelihood of for , we utilize more aggressive hindsight rewards, so that a large positive reward given to a successful hindsight transition greatly increases the Q-value of the hindsight state-action pair, which indirectly drives an aggressive policy update towards choosing this maximizing action for the given hindsight state.

We test our hypothesis by introducing two real-valued scalar multipliers, and , to distinguish between real rewards and hindsight rewards as follows:

(11)
(12)

where r() is the given reward function for the task.

We refer to and as trade-off parameters as they proportionally increase the value of one category of reward with respect to the other. We investigate the impact of this weighted reward mechanism on finding the optimal policy. Vanilla HER is a special case with .

Figure 1: Illustration of the two environments: Reacher (Top) and Finger (Bottom). See text for details.

ARCHER framework requires that . Hence using Eqn. (11) and Eqn. (12) we get,

(13)

Therefore in domains with positive reward functions, i.e. , ARCHER comprises trade-offs with . Conversely, in negative reward functions, i.e. , ARCHER comprises with trade-offs .

Using trade-off, the target values for real experience and hindsight experience (Eqn. 7) can be rewritten as,

(14)
(15)

4 Experiments

In this section, we present our experimental analyses and ablation studies, along with the corresponding inferences.

4.1 Simulation Environments

We evaluate our method on the DeepMind (DM) Control Suite [Tassa et al.2018] simulation software. This library consists of a set of continuous control environments in Python, built on top of the MuJoCo physics engine [Todorov, Erez, and Tassa2012]. Each environment in the suite provides a physics task along with a well-defined continuous action space , continuous state/observation space , and intrinsic transition dynamics based on the physics engine. For our experiments, we program our own reward functions to conduct ablation studies on ARCHER and verify its robustness, as detailed in the following sections. We tested our algorithm on the following two domains:

Reacher

The Reacher environment (Fig. 1 (a)) includes a 2-DoF planar robot arm where the agent must place its end effector at a randomized target, indicated by the red sphere. The state input is a 4-element vector with the first 2 elements containing the generalized positions and next 2 elements containing the generalized velocities of the two arm joints, as encoded by the Mujoco physics engine. is concatenated with a real/hindsight goal which specifies the 3-dimensional global coordinate of the target sphere/end effector respectively. The action is a 2-dimensional vector where each element informs the relative displacement of each joint from its current position.

Finger

The finger environment (Fig. 1 (b)) is a multi-body arrangement where a planar 2-DoF robot arm has to flick a spinner resting on an unactuated hinge so as to place the red tip of the spinner on the target indicated by a red sphere. The arm must therefore learn a policy in a environment with discontinuous dynamics [Tassa and Todorov2010]. The state is concatenation of a 4-dimensional position vector (generalized joint points as the first two elements and the relative (x, z) position of the spinner tip to its hinge as the next two elements), a 3-dimensional velocity vector of the 3 joints and followed by signals from the two touch-sensors on the top and bottom of the spinner. This 9-dimensional state is concatenated with a goal determined by the position of target sphere relative to the hinge for real episodes, and the relative position of the spinner tip for hindsight episodes. The 2-dimensional action vector specifies the relative displacement of the two arm joint from their current positions.

In both domains, the initial state and target locations are randomly initialized at the beginning of every episode.

4.2 Reward Functions

We analyze the performance of ARCHER in comparison to vanilla HER, across 3 types of rewards for each task:

  1. Binary -1/0 reward: In this case, the agent receives a reward of -1 for every time-step in the episode where the goal is not achieved, and receives 0 when the agent it successful.

    (16)

    This reward function penalizes the agent for not achieving the goal at every time-step and therefore the agent is incentivized to learn a time-efficient optimal policy. The negative reward punishes the agent for executing unproductive actions. It is used in the vanilla HER.

  2. Binary 0/+1 reward: In this case, the agent is awarded a value of 0 for every unsuccessful time-step and is granted a reward of 1 when the goal is achieved.

    (17)

    The positive rewards encourage the agent to learn the policy which allows it to collect most reward and encourages the successful actions taken by the agent.

  3. Shaped reward: In this case, the agent is provided with a continuous real-valued reward signal, in proportion to some metric representing how close to/far away from success the agent is. For our experiments, we have selected the negative of Frobenius norm of the difference vector between the achieved goal and actual goal. The Frobenius norm of a vector is given as

    Therefore, the shaped reward function is given by

    (18)

    The agent receives a large negative reward for states far away from the goal state and rewards closer to 0 when the agent is close to the goal. Vanilla HER performs poorly in tasks with shaped reward.

Figure 2: Policy performance in the Reacher (Top) and the Finger (Bottom) environment with sparse binary negative rewards and the final sampling strategy for hindsight goals. Best viewed in color.
Figure 3: Ablation studies in the Reacher (Top) and the Finger (Bottom) environment. See text for detail. Best viewed in color.

Strategy for sampling goals for experience replay: We test two strategies proposed in HER [Andrychowicz et al.2017]: first is the final strategy which replays with the hindsight goal corresponding to the final state in each episode, and the other is the future strategy which replays with random states which come from the same episode as the transition being replayed and were observed after it (we use the best reported practice with ).

4.3 Trade-off Parameters

For each of our experiments, we carefully selected the trade-off parameters to gain insight into the relative reward optimization between hindsight and standard experience replay. The following set of weights helps us understand the impact on performance driven by (i) the ratio between hindsight and real reward weights (ii) the magnitude of these weights relative to baseline HER.

  1. : These weights are the same as vanilla-HER, which serves as the baseline standard for our tests.

  2. : The magnitude of the smaller weight is half of the baseline weight. We have and .

  3. : The magnitude of the larger weight is twice the baseline weight. We have and .

  4. : The magnitude of the larger weight is twice the baseline weight and the magnitude of the smaller weight is half of the baseline weight. We have and .

4.4 Network Architecture and Training

For our experimental setup, the actor and critic networks were designed with 2 fully connected hidden layers, consisting of 400 ReLU 

[Nair and Hinton2010]neurons in the first and 300 ReLU neurons in the second layer. The output layer of the actor networks used activation. The layers of the networks were initialized uniformly from where was the fan-in of the layer. The final layers of the networks were initialized uniformly in the range . The discount factor was 0.98. The weight for the weighted update of the target networks was 0.001. To encourage exploration, we added Ornstein-Uhlenbeck process [Uhlenbeck and Ornstein1930] noise with and to the output of the actor. The added noise is multiplied by a factor , with an initial value of 0.1 with an exponential decay factor of 0.99, to gradually reduce exploration. The learning rates for the actor and critic were and respectively, and the networks were trained using Adam [Kingma and Ba2014] optimization. We used a replay buffer of size from which minibatches of size 128 were uniformly sampled for training. The networks were trained for 2000 cycles, where 1 cycle represents running the policy for 16 episodes followed by 40 steps of optimization.

4.5 Results

We evaluated our method by comparing the success rate of the learned policies against the required number of cycles to achieve that performance. An episode was considered successful if on final step of the episode, the end effector was placed at the target in the case of Reacher; or the spinner tip was aligned with the target in the case of Spinner. Each episode consisted of 50 MuJoCo time-steps. The results presented in Fig. 2 and Fig. 3 show the performance curves of the actual actor networks of each agent, averaged over 5 random seeds and smoothed across the past 50 elements. The blue tinted plots depict HER with tradeoff while the orange/red tinted plots depict HER with tradeoff. Vanilla HER is shown with a perforated black plot.

Experiment I: Using Sparse Negative Rewards and the Final Goal Sampling Strategy.

Fig. 2 displays the final performance plots in the Reacher and Finger domains with a binary negative -1/0 reward function (16). In both domains, we observe from the leftmost graph that vanilla HER greatly improves the performance of DDPG. We also see that learning exclusively from only real or only hindsight experiences decreases performance. Hence we need a combination of real and hindsight experiences for competent learning. The following 3 graphs show that the fastest performance is demonstrated in the curves where , representing ARCHER. Moreoever, when we specify trade-offs to exacerbate the discrepancy between the true hindsight probability and biased probability (), performance is adversely affected and sample-efficiency decreases. This confirms our hypothesis of hindsight bias in HER.

Experiment II: Ablation Studies

Fig. 3 shows the results of our ablation studies.

  1. The first column shows the performance curves for the different algorithms when presented with a sparse binary positive 0/+1 reward function (17). The striking observation is that the orange tinted curves are above the baseline while the blue curves fall below. This result is consistent with ARCHER as when the reward function is positive, ensures that hindsight rewards are numerically greater. This graph shows that ARCHER is not just a coincidence of implicit hyper-parameter tuning such as learning rate increase, but is robust to changes in reward sign.

  2. In the second column we illustrate the effectiveness of ARCHER in dense binary negative reward condition, where the size of the target sphere in both domains is magnified. As shown in the graph, the blue plots depicting ARCHER perform better than vanilla HER. In Reacher, the other curves eventually catch up to ARCHER but in the high-dimensional Finger domain, ARCHER maintains a noticeable lead. Hence the main benefit of ARCHER lies in high-dimensional, sparse reward domains.

  3. In the third column, we plot the results for vanilla HER and HER with trade-offs when 4 ”future” goals are sampled for hindsight replay with negative binary sparse reward. In the fourth column, we show the performance of the different algorithms when provided negative shaped reward (18). Under both arrangements, we observe that the blue ARCHER curves deliver the most sample efficiency. Shaped reward was mentioned as a weakness for vanilla HER in  [Andrychowicz et al.2017]. Here, we see that ARCHER does outperform the baseline even with shaped reward, although the final performance is lower than the other reward functions.

5 Conclusion and Discussion

In this paper, we identify bias in hindsight experience replay and present a method ARCHER to counter the bias and increase sample-efficiency of deep RL algorithms, using numerically greater hindsight rewards. We also empirically verified our hypothesis of hindsight bias in vanilla HER by exhibiting that when DDPG is trained with rewards opposite to those specified by ARCHER, the bias is amplified and performance degrades. Using the Finger and Reacher simulation environments from DeepMind control suite, we demonstrated the increased sample efficiency derived from ARCHER in comparison to baseline vanilla HER. Our ablation studies prove ARCHER consistently outperforms vanilla HER across various reward functions and task complexities. Hence, ARCHER grants reliable sample-efficiency especially in continuous control and robotic with limited computing budget.

A few interesting directions emerge for further exploration. Some of our experiments reveal that ARCHER enjoys higher sample-efficiency only until a context-dependent number of samples, after which vanilla HER catches up to ARCHER. This effect makes intuitive sense as the high performance of ARCHER leads to the fast convergence of real and hindsight experiences, and diminished hindsight bias. Hence, a scheduled annealing of ARCHER remains of interest. Also, we specifically constructed a simple linear relation to derive a more informative hindsight reward function, however we believe that there may exists a more complex mapping between real and hindsight rewards and hence it may be advantageous to introduce a generative model to learn the latent mapping. Moving beyond predefined trade-off parameters and limited number of their combinations will allow us to construct a more mathematically rigorous theory of hindsight bias. We did not deploy ARCHER to real robots, and will investigate this in the future.

References