Log In Sign Up

Experience Replay Using Transition Sequences

by   Thommen George Karimpanal, et al.
Singapore University of Technology and Design

Experience replay is one of the most commonly used approaches to improve the sample efficiency of reinforcement learning algorithms. In this work, we propose an approach to select and replay sequences of transitions in order to accelerate the learning of a reinforcement learning agent in an off-policy setting. In addition to selecting appropriate sequences, we also artificially construct transition sequences using information gathered from previous agent-environment interactions. These sequences, when replayed, allow value function information to trickle down to larger sections of the state/state-action space, thereby making the most of the agent's experience. We demonstrate our approach on modified versions of standard reinforcement learning tasks such as the mountain car and puddle world problems and empirically show that it enables better learning of value functions as compared to other forms of experience replay. Further, we briefly discuss some of the possible extensions to this work, as well as applications and situations where this approach could be particularly useful.


page 1

page 2

page 3

page 4


Time Limits in Reinforcement Learning

In reinforcement learning, it is common to let an agent interact for a f...

Organizing Experience: A Deeper Look at Replay Mechanisms for Sample-based Planning in Continuous State Domains

Model-based strategies for control are critical to obtain sample efficie...

When to use parametric models in reinforcement learning?

We examine the question of when and how parametric models are most usefu...

The Benefits of Model-Based Generalization in Reinforcement Learning

Model-Based Reinforcement Learning (RL) is widely believed to have the p...

Paused Agent Replay Refresh

Reinforcement learning algorithms have become more complex since the inv...

1 Introduction

Real-world artificial agents ideally need to be able to learn as much as possible from their interactions with the environment. This is especially true for mobile robots operating within the reinforcement learning (RL) framework, where the cost of acquiring information from the environment through exploration generally exceeds the computational cost of learning (1; 2; 3; 4).

Experience replay (5) is a technique that reuses information gathered from past experiences to improve the efficiency of learning. In order to replay stored experiences using this approach, an off-policy (6; 7) setting is a prerequisite. In off-policy learning, the policy that dictates the agent’s current actions is referred to as the behavior policy. Other policies—possibly corresponding to other tasks—that the agent aims to learn using actions arising from the behavior policy are referred to as target policies. Off-policy algorithms utilize the agent’s behavior policy to interact with the environment, while simultaneously updating the value functions associated with the target policies. These algorithms can hence be used to parallelize learning, and, thus gather as much knowledge as possible using real experiences (8; 9; 10)

. However, when the behavior and target policies differ considerably from each other, the actions executed by the behavior policy may only seldom correspond to those of the target policy. This could lead to poor estimates of the corresponding value function. In such cases, and generally, in environments where desirable experiences are rare occurrences, experience replay could be employed to improve the estimates by storing and replaying transitions (state, actions and rewards) from time to time.

Although most experience replay approaches store and reuse individual transitions, replaying sequences of transitions may have certain advantages. For instance, if a value function update following a particular transition results in a relatively large change in the value of the corresponding state or state-action pair, this change will have a considerable influence on the bootstrapping targets of states or state-action pairs that led to this transition. Hence, the effects of this change should ideally be propagated to these states or state-action pairs. If instead of individual transitions, sequences of transitions are replayed, this propagation can be achieved in a straightforward manner. Our approach aims to improve the efficiency of learning by replaying transition sequences in this manner. The sequences are selected on the basis of the magnitudes of the temporal difference (TD) errors associated with them. We hypothesize that selecting sequences that contain transitions associated with higher magnitudes of TD errors allow considerable learning progress to take place. This is enabled by the propagation of the effects of these errors to the values associated with other states or state-action pairs in the transition sequence.

Replaying a greater variety of such sequences would result in better propagation of the mentioned effects to other regions in the state/state-action space. Hence, in order to aid the propagation in this manner, other sequences that could have occurred are artificially constructed by comparing the state trajectories of previously observed sequences. These virtual transition sequences are appended to the replay memory, and they help bring about learning progress in other regions of the state/state-action space when replayed.

Figure 1: Structure of the proposed algorithm in contrast to the traditional off-policy structure. and denote the action-value function and reward respectively.

The generated transition sequences are virtual in the sense that they may have never occurred in reality, but are constructed from sequences that have actually occurred in the past. The additional replay updates corresponding to the mentioned transition sequences supplement the regular off-policy value function updates that follow the real-world execution of actions, thereby making the most out of the agent’s interactions with the environment.

2 Background

The problem of learning from limited experience is not new in the field of RL (11; 12). Generally, learning speed and sample efficiency are critical factors that determine the feasibility of deploying learning algorithms in the real world. Particularly for robotics applications, these factors are even more important, as exploration of the environment is typically time and energy expensive (13; 14). It is thus important for a learning agent to be able to gather as much relevant knowledge as possible from whatever exploratory actions occur.

Off-policy algorithms are well suited to this need as it enables multiple value functions to be learned together in parallel. When the behavior and target policies vary considerably from each other, importance sampling (15; 6)

is commonly used in order to obtain better estimates of the value functions. Importance sampling reduces the variance of the estimate by taking into account the distributions associated with the behavior and target policies, and making modifications to the off-policy update equations accordingly. However, the estimates are still unlikely to be close to their optimal values if the agent receives very little experience relevant to a particular task.

Experience replay can be a useful approach to deal with such situations where favorable experiences occur rarely. Here, a replay memory is maintained, which consists of previously experienced states and actions, and the corresponding subsequent states and rewards observed following the execution of those actions. The information contained in the replay memory is used from time to time in order to update the value functions, thereby making better use of the agent-environment interactions. This approach has received a lot of attention in recent years due to its utility in deep RL applications (16; 17; 18; 2; 19).

It is known that for experience replay, certain transitions may be more useful than others. Recent works (3; 20) have explored different ways in which transitions may be prioritized. Schaul et al. (3) prioritized transitions on the basis of their associated TD errors. They also briefly mentioned the possibility of replaying transitions in a sequential manner. The experience replay framework developed by Adam et al. (2) involved some variants that replayed sequences of experiences, but these sequences were drawn randomly from the replay memory.

As in Schaul et al. (3), TD errors have been frequently used as a basis for prioritization in other RL problems (21; 22; 3). In particular, the model-based approach of prioritized sweeping (23; 24) prioritizes backups that are expected to result in a significant change in the value function.

The algorithm we propose here is model-free, and it is based on the idea of selectively reusing previous experience. However, we describe the reuse of sequences of transitions based on the TD errors observed when these transitions take place. Replaying sequences of experiences also seems to be biologically plausible (25; 26). In addition, it is known that animals tend to remember experiences that lead to high rewards (27). This is an idea reflected in our work, as only those transition sequences that lead to high rewards are considered for being stored in the replay memory. In filtering transition sequences in this manner, we simultaneously address the issue of determining which experiences are to be stored.

In addition to selecting transition sequences, we also generate virtual sequences of transitions which the agent could have possibly experienced, but in reality, did not. This virtual experience is then replayed to improve the agent’s learning. Some early approaches in RL, such as the dyna architecture (28)

also made use of simulated experience to improve the value function estimates. However, unlike the approach proposed here, the simulated experience was generated based on models of the reward function and transition probabilities which were continuously updated based on the agent’s interactions with the environment. In this sense, the virtual experience generated in our approach is more grounded in reality, as it is based directly on the data collected through the agent-environment interaction. Our approach also recognizes the real-world limitations of replay memory

(19), and stores only a certain amount of information at a time, specified by memory parameters. The selected and generated sequences are stored in the replay memory in the form of libraries which are continuously updated so that the agent is equipped with transition sequences that are most relevant to the task at hand.

3 Methodology

RL agents are designed to learn from experiences gathered by interacting with their environment. However, when certain critical experiences occur rarely, their effect on the agent’s learning is limited, and this adversely affects the agent’s value function estimates. For example, in the context of off-policy learning, value functions may converge poorly if the target and behavior policies are vastly different from one another. The approach proposed here improves the learning of value functions in such cases by selectively replaying sequences of experiences. Similar to ordinary experience replay, this approach requires additional memory and computation. However, this can be justified for real-world robotics applications where the time and energy costs of exploration typically outweigh the cost of memory storage and the accompanying increase in computational requirements (e.g. (29)). We describe our approach in the context of learning target policies as mentioned above, in a non-ideal situation where the behavior and target policies are substantially different.

The idea of selecting appropriate transition sequences for replay is relatively straightforward. In order to improve the agent’s learning, first, we simply keep track of the state, actions, rewards and TD errors associated with each transition. Generally, high rewards occur rarely, so when such an event is observed, we consider storing the corresponding sequence of transitions into a replay library . In this manner, we use the reward information as a means to filter transition sequences. The approach is similar to that used by Narasimhan et al. (20), where transitions associated with positive rewards are prioritized for replay.

Among the transition sequences considered for inclusion in the library , those containing transitions with high absolute TD error values are considered to be the ones with high potential for learning progress. Hence, they are accordingly prioritized for replay. The key idea is that when the TD error associated with a particular transition is large in magnitude, it generally implies a proportionately greater change in the value of the corresponding state/state-action pair. Such large changes have the potential to influence the values of the states/state-action pairs leading to it, which implies a high potential for learning. Hence, prioritizing such sequences of transitions for replay is likely to bring about greater learning progress. Transition sequences associated with large magnitudes of TD error are retained in the library, while those with lower magnitudes are removed and replaced with better alternatives. In reality, such transition sequences may be very long and hence, impractical to store. Due to such practical considerations, we store only a portion of the sequence, based on a predetermined memory parameter. The library is continuously updated as and when the agent-environment interaction takes place, such that it will eventually contain sequences associated with the highest absolute TD errors.

As described earlier, replaying suitable sequences allows the effects of large changes in value functions to be propagated throughout the sequence. In order to propagate this information even further to other regions of the state/state-action space, we use the sequences in to construct additional transition sequences which could have possibly occurred. These virtual sequences are stored in another library , and later used for experience replay.

Figure 2: (a) Trajectories corresponding to two hypothetical behavior policies are shown. A portion of the trajectory associated with a high reward (and stored in ) is highlighted (b) The virtual trajectory constructed from the two behavior policies is highlighted. The states, actions and rewards associated with this trajectory constitute a virtual transition sequence.

In order to intuitively describe our approach of artificially constructing sequences, we consider the hypothetical example shown in Figure 1(a), where an agent executes behavior policies that help it learn to navigate towards location from the start location. However, using off-policy learning, we aim to learn value functions corresponding to the policy that helps the agent navigate towards location .

The trajectories shown in Figure 1(a) correspond to hypothetical actions dictated by the behavior policy midway through the learning process, during two separate episodes. The trajectories begin at the start location and terminate at location . However, the trajectory corresponding to behavior policy also happens to pass through location , at which point the agent receives a high reward. This triggers the transition sequence storage mechanism described earlier, and we assume that some portion of the sequence (shown by the highlighted portion of the trajectory in Figure 1(a)) is stored in library . Behavior policy takes the agent directly from the start location towards the location , where it terminates. As the agent moves along its trajectory, it intersects with the state trajectory corresponding to the sequence stored in . Using this intersection, it is possible to artificially construct additional trajectories (and their associated transition sequences) that are successful with respect to the task of navigating to location . The highlighted portions of the trajectories corresponding to the two behavior policies in Figure 1(b) show such a state trajectory, constructed using information related to the intersection of portions of the two previously observed trajectories. The state, action and reward sequences associated with this highlighted trajectory form a virtual transition sequence.

Such artificially constructed transition sequences present the possibility of considerable learning progress. This is because, when replayed, they help propagate the large learning potential (characterized by large magnitudes of TD errors) associated with sequences in to other regions of the state/state-action space. These replay updates supplement the off-policy value function updates that are carried out in parallel, thus accelerating the learning of the task in question. This outlines the basic idea behind our approach.

Fundamentally, our approach can be decomposed into three steps:

  1. Tracking and storage of relevant transition sequences

  2. Construction of virtual transition sequences using the stored transition sequences

  3. Replaying the transition sequences

These steps are explained in detail in Sections 3.1, 3.2 and 3.3.

3.1 Tracking and Storage of Relevant Transition Sequences

As described, virtual transition sequences are constructed by joining together two transition sequences. One of them, say , composed of transitions, is historically successful—it has experienced high rewards—with respect to the task, and is part of the library . The other sequence, , is simply a sequence of the latest transitions executed by the agent.

If the agent starts at state and moves through intermediate states and eventually to (most recent state) by executing a series of actions , it receives rewards from the environment. These transitions comprise the transition sequence .



We respectively refer to , and as the state, action and reward transition sequences between states and .

For the case of the transition sequence , we keep track of the sequence of TD errors observed as well. If a high reward is observed in transition , then:


where .

The memory parameters and are chosen based on the memory constraints of the agent. They determine how much of the recent agent-environment interaction history is to be stored in memory.

It is possible that the agent encounters a number of transitions associated with high rewards while executing the behavior policy. Accordingly, a number of such successful transition sequences may also exist. These sequences are maintained in the library in a manner similar to the Policy Library through Policy Reuse (PLPR) algorithm (30). To decide whether to include a new transition sequence into the library , we determine the maximum absolute value of the TD error sequence corresponding to and check whether it is -close—the parameter determines the exclusivity of the library—to the maximum of the corresponding values associated with the transition sequences in . If this is the case, then is included in . Using the absolute TD error as a basis for selection, we maintain a fixed number () of transition sequences in the library . This ensures that the library is continuously updated with the latest transition sequences associated with the highest absolute TD errors. The complete algorithm is illustrated in Algorithm 1.

2: Parameter that determines the exclusivity of the library
3: Parameter that determines the number of transition sequences allowed in the library
4: Sequence of TD errors corresponding to a transition sequence
5: A library of transition sequences ()
6: New transition sequence to be evaluated
8:for  do
9:   )
10:end for
11:if  then
13:   Number of transition sequences in
14:   if  then
16:   end if
17:end if
Algorithm 1 Maintaining a replay library of transition sequences

3.2 Virtual Transition Sequences

Once the transition sequence is available and a library of successful transition sequences is obtained, we use this information to construct a library of virtual transition sequences . The virtual transition sequences are constructed by first finding points of intersection in the state transition sequences of and the ’s in .

Let us consider the transition sequence :

and a transition sequence :
Let be a subset of such that:
If , then:

Once points of intersection have been obtained as described above, each of the two sequences and are decomposed into two subsequences at the point of intersection such that:


where and ,


where and . The virtual transition sequence is then simply:


We perform the above procedure for each transition sequence in to obtain the corresponding virtual transition sequences . These virtual transition sequences are stored in a library :

where denotes the number of virtual transition sequences in , subjected to the constraint .

Once the library has been constructed, we replay the sequences contained in it to improve the estimates of the value function. The details of this are discussed in Section 3.3.

3.3 Replaying the Transition Sequences

In order to make use of the transition sequences described, each of the state-action-reward triads in the transition sequence is replayed as if the agent had actually experienced them. To achieve a quicker propagation of information, the sequences are replayed in the reverse temporal order in which they were observed, as shown in Algorithm 2. Such an idea has been briefly mentioned in previous studies on experience replay (3; 2). In addition, a similar mechanism of reverse replay has been observed in animals (31).

Similarly, sequences in may also be replayed from time to time. Replaying sequences from and in this manner causes the effects of large absolute TD errors originating from further up in the sequence to propagate through the respective transitions, ultimately leading to better estimates of the value function. The transitions are replayed as per the standard -learning update equation shown below:


Where and refer to the state and action at transition , and and represent the action-value function and reward corresponding to the task. The variable is a bound variable that represents any action in the action set . The learning rate and discount parameters are represented by and respectively.

The sequence in Equation (5) is a subset of , which is in turn part of the library and thus associated with a high absolute TD error. When replaying , the effects of the high absolute TD errors propagate from the values of state/state-action pairs in to those in . Hence, in case of multiple points of intersection, we consider points that are furthest down . In other words, the intersection point is chosen to maximize the length of . In this manner, a larger number of state-action values experience improvements brought about by replaying the transition sequences.

2: learning rate
3: discount factor
4: A library of virtual transition sequences with sequences
5:for  do
6:   number of triads in
8:   while  do
11:   end while
12:end for
Algorithm 2 Replay of virtual transition sequences from library

4 Results and Discussion

We demonstrate our approach on modified versions of two standard reinforcement learning tasks. The first is a navigation/puddle-world problem (Figure 3), and the second is a mountain car problem (Figure 5). In both these problems, behavior policies are generated to solve a given task (which we refer to as the primary task) relatively greedily, while the value function for another task of interest (which we refer to as the secondary task) is simultaneously learned in an off-policy manner. The secondary task is intentionally made more difficult by making appropriate modifications to the environment. Such a setting best demonstrates the effectiveness of our approach and emphasizes its advantages over other experience replay approaches. We characterize the difficulty of the secondary task with a difficulty ratio , which is the fraction of the executed behavior policies that experience a high reward with respect to the secondary task. A low value of indicates that achieving the secondary task under the given behavior policy is difficult.

4.1 Navigation/Puddle-World Task

Figure 3: Navigation environment used to demonstrate the approach of replaying transition sequences

In the navigation environment, the simulated agent is assigned tasks of navigating to certain locations in its environment. We consider two locations, and , which represent the primary and secondary task locations respectively. The environment is set up such that the location corresponding to high rewards with respect to the secondary task lies far away from that of the primary task (see Figure 3). In addition to this, the accessibility to the secondary task location is deliberately limited by surrounding it with obstacles on all but one side. These modifications contribute towards a low value of , especially when the agent operates with a greedy behavior policy with respect to the primary task.

The agent is assumed to be able to sense its location in the environment accurately and is capable of detecting obstacles directly in front of it, up to 1 unit away. It can move around in the environment at a maximum speed of 1 unit per time step by executing actions to take it forwards, backwards, sideways and diagonally forwards or backwards to either side. In addition to these actions, the agent can choose to hold its current position. However, the transitions resulting from these actions are probabilistic in nature. The intended movements occur only 80 % of the time, and for the remaining 20 %, the - and -coordinates may deviate from their intended values by 1 unit. Also, the agent’s location does not change if the chosen action forces it to run into an obstacle.

The agent employs tabular -learning with a relatively greedy policy () that attempts to maximize the expected sum of primary rewards. The reward structure for both tasks is such that the agent receives a high reward () for visiting the respective goal locations, and a high penalty () for bumping into an obstacle in the environment. In addition to this, the agent is assigned a living penalty () for each action that fails to result in the goal state. In all simulations, the discount factor is set to be and the learning rate is set to a value of . These values were found to work well in both the navigation as well as the mountain-car environments.

Figure 4: Comparison of the average secondary returns using different experience replay approaches as well as -learning without experience replay in the navigation environment. For the different experience replay approaches, the number of replay updates are controlled to be the same.

In the environment described, the agent executes actions to learn the primary task. Simultaneously, the approach described in Section 3 is employed to learn the value functions associated with the secondary task. At each episode of the learning process, the agent’s performance with respect to the secondary task is evaluated. This is done by allowing the agent to execute actions from randomly generated starting points in the state space and measuring the average sum of rewards (return) accumulated per episode.

Figure 4 shows the average return for the secondary task plotted for runs of learning episodes using different learning approaches. The low average value of ( as indicated in Figure 4) indicates the relatively high difficulty of the secondary task under the behavior policy being executed. As observed in Figure 4, an agent that replays transition sequences manages to accumulate high average returns at a much faster rate as compared to regular -learning. The approach also performs better than other experience replay approaches for the same number of replay updates.

10 1559.7
100 2509.7
1000 2610.4
10 1072.5
100 1159.2
1000 2610.4
10 2236.6
50 2610.4
100 2679.5
  • With regular -learning (without experience replay),

Table 1: Average secondary returns accumulated per episode () using different values of the memory parameters in the navigation environment

Table 1 shows the average return for the secondary task accumulated per episode () during runs of the navigation task for different values of memory parameters , and used in our approach. Each of the parameters are varied separately while keeping the other parameters fixed to their default values. The default values used for , and are , and respectively.

4.2 Mountain Car Task

Figure 5: Mountain car environment used to demonstrate off-policy learning using virtual transition sequences

In the mountain car task, the agent, an under-powered vehicle represented by the circle in Figure 5 is assigned a primary task of getting out of the trough and visiting point . The act of visiting point is treated as the secondary task. The agent is assigned a high reward () for for fulfilling the respective objectives, and a living penalty () is assigned for all other situations. At each time step, the agent can choose from three possible actions: (1) accelerating in the positive direction, (2) accelerating in the negative direction, and (3) applying no control. The environment is discretized such that unique positions and unique velocity values are possible.

The mountain is described by the equation such that point is higher than . Also, the average slope leading to is steeper than that leading to . In addition to this, the agent is set to be relatively greedy with respect to the primary task, with an exploration parameter . These factors make the secondary task more difficult, resulting in a low value of () under the policy executed.

Figure 6 shows the average secondary task returns for runs of learning episodes. It is seen that especially during the initial phase of learning, the agent accumulates rewards at a higher rate as compared to other learning approaches. As in the navigation task, the number of replay updates are restricted to be the same while comparing the different experience replay approaches in Figure 6. Analogous to Table 1, Table 2 shows the average secondary returns accumulated per episode () over runs in the mountain-car environment, for different values of the memory parameters. The default values for , and are the same as those mentioned in the navigation environment, that is, , and respectively.

Figure 6: Comparison of the average secondary returns using different experience replay approaches as well as -learning without experience replay in the mountain-car environment. For the different experience replay approaches, the number of replay updates are controlled to be the same.
10 221.0
100 225.1
1000 229.9
10 129.9
100 190.5
1000 229.9
10 225.6
50 229.9
100 228.4
  • With regular -learning (without experience replay),

Table 2: Average secondary returns accumulated per episode () using different values of the memory parameters in the mountain car environment

From Figures 4 and 6, the agent is seen to be able to accumulate significantly higher average secondary returns per episode when experiences are replayed. Among the experience replay approaches, the approach of replaying transition sequences is superior for the same number of replay updates. This is especially true in the navigation environment, where visits to regions associated with high secondary task rewards are much rarer, as indicated by the low value of . In the mountain car problem, the visits are more frequent, and the differences between the different experience replay approaches are less significant. In both environments, the performances of the approaches that replay individual transitions—experience replay with uniform random sampling and prioritized experience replay—are found to be nearly equivalent.

The approach of replaying transition sequences seems to be particularly sensitive to the memory parameter , with better performances being achieved for larger values of . A possible explanation for this could simply be that larger values of correspond to longer sequences, which allow a larger number of replay updates to occur in more regions of the state/state-action space. The influence of the length of the sequence, specified by the parameter is also similar in nature, but its impact on the performance is less emphatic. This could be because longer sequences allow a greater chance for their state trajectories to intersect with those of , thus improving the chances of virtual transition sequences being discovered, and of the agent’s value functions being updated using virtual experiences. However, the parameter , associated with the size of the library does not seem to have a noticeable influence on the performance of this approach. This is probably due to the fact that the library (and consequently ) is continuously updated with new, suitable transition sequences (successful sequences associated with higher magnitudes of TD errors) as and when they are observed. Hence, the storage of a number of transition sequences in the libraries becomes largely redundant.

Although the method of constructing virtual transition sequences is more naturally applicable to the tabular case, it could also possibly be extended to approaches with function approximation. However, soft intersections between state trajectories would have to be considered instead of absolute intersections. That is, while comparing the state trajectories and , the existence of could be considered if it is close to elements in both and within some specified tolerance limit. One of the limitations of constructing virtual transition sequences is that in higher dimensional spaces, intersections in the state trajectories may become less frequent. However, other sequences in the library may still be replayed. If appropriate sequences have not yet been discovered or constructed, and are thus not available for replay, other experience replay approaches that replay individual transitions can be used to accelerate learning in the meanwhile.

Perhaps another limitation of the approach described here is that constructing the library requires some notion of a goal state associated with high rewards. By tracking the statistical properties such as the mean and variance of the rewards experienced by an agent in its environment in an online manner, the notion of what qualifies as a high reward could be automated using suitable thresholds. In addition to this, other criteria such as the returns or average absolute TD errors of a sequence could also be used to maintain the library.

It is worth adding that the memory parameters , and have been set arbitrarily in the examples described here. Selecting appropriate values for these parameters as the agent interacts with its environment could be a topic for further research.

The approach of replaying transition sequences has direct applications in multi-task RL, where agents are required to learn multiple tasks in parallel. In addition, this approach could also be significantly useful in deep RL applications. Certain tasks could be associated with the occurrence of relatively rare events during the agent’s interaction with the environment. The replay of virtual transition sequences could further improve the learning in such tasks. This approach could also prove to be particularly valuable in fields such as robotics, where exploration of the state/state-action space is typically expensive in terms of time and energy. By reusing the agent-environment interactions in the manner described here, reasonably good estimates of the value functions corresponding to multiple tasks can be maintained, thereby improving the efficiency of exploration.

5 Conclusion

In this work, we described an approach to replay sequences of transitions to accelerate the learning of tasks in an off-policy setting. Suitable transition sequences are selected and stored in a replay library based on the magnitudes of the TD errors associated with them. Using these sequences, we showed that it is possible to construct virtual experiences in the form of virtual transition sequences, which could be replayed to improve an agent’s learning, especially in environments where desirable events occur rarely. We demonstrated the benefits of this approach by applying it to versions of standard reinforcement learning tasks such as the puddle-world and mountain-car tasks. In both tasks, a significant improvement in learning speed was observed compared to regular -learning as well as other forms of experience replay. Further, the influence of the different memory parameters used was described and evaluated empirically, and possible extensions to this work were briefly discussed. Characterized by controllable memory parameters and the potential to significantly improve the efficiency of exploration at the expense of some increase in computation, the approach of using replaying transition sequences could be especially useful in fields such as robotics, where these factors are of prime importance.


The authors thank Richard S. Sutton from the University of Alberta for his feedback and many helpful discussions during the development of this work.



  • (1) Z. Wang, V. Bapst, N. Heess, V. Mnih, R. Munos, K. Kavukcuoglu, N. de Freitas, Sample efficient actor-critic with experience replay, arXiv preprint arXiv:1611.01224.
  • (2) S. Adam, L. Busoniu, R. Babuska, Experience replay for real-time reinforcement learning control, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 42 (2) (2012) 201–212.
  • (3) T. Schaul, J. Quan, I. Antonoglou, D. Silver, Prioritized experience replay, in: International Conference on Learning Representations, Puerto Rico, 2016, p. 1.
  • (4) T. G. Karimpanal, E. Wilhelm, Identification and off-policy learning of multiple objectives using adaptive clustering, Neurocomputing (In Press) (2017) arXiv:1705.06342 [cs.AI].
  • (5)

    L.-J. Lin, Self-improving reactive agents based on reinforcement learning, planning and teaching, Machine Learning 8 (3-4) (1992) 293–321.

  • (6) R. S. Sutton, A. G. Barto, Reinforcement learning: An introduction (2011).
  • (7) M. Geist, B. Scherrer, et al., Off-policy learning with eligibility traces: a survey., Journal of Machine Learning Research 15 (1) (2014) 289–333.
  • (8) R. S. Sutton, J. Modayil, M. Delp, T. Degris, P. M. Pilarski, A. White, D. Precup, Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction, in: The 10th International Conference on Autonomous Agents and Multiagent Systems-Volume 2, International Foundation for Autonomous Agents and Multiagent Systems, 2011, pp. 761–768.
  • (9) A. White, J. Modayil, R. S. Sutton, Scaling life-long off-policy learning, in: Development and Learning and Epigenetic Robotics (ICDL), 2012 IEEE International Conference on, IEEE, 2012, pp. 1–6.
  • (10) J. Modayil, A. White, R. S. Sutton, Multi-timescale nexting in a reinforcement learning robot, Adaptive Behavior 22 (2) (2014) 146–160.
  • (11) S. B. Thrun, Efficient exploration in reinforcement learning, Tech. rep., Pittsburgh, PA, USA (1992).
  • (12) P. S. Thomas, E. Brunskill, Data-efficient off-policy policy evaluation for reinforcement learning, in: International Conference on Machine Learning, 2016.
  • (13) B. Bakker, V. Zhumatiy, G. Gruener, J. Schmidhuber, Quasi-online reinforcement learning for robots, in: Robotics and Automation, 2006. ICRA 2006. Proceedings 2006 IEEE International Conference on, IEEE, 2006, pp. 2997–3002.
  • (14) J. Kober, J. A. Bagnell, J. Peters, Reinforcement learning in robotics: A survey, The International Journal of Robotics Research 32 (11) (2013) 1238–1274.
  • (15) R. Y. Rubinstein, D. P. Kroese, Simulation and the Monte Carlo method, John Wiley & Sons, 2016.
  • (16) V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al., Human-level control through deep reinforcement learning, Nature 518 (7540) (2015) 529–533.
  • (17) V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. P. Lillicrap, T. Harley, D. Silver, K. Kavukcuoglu, Asynchronous methods for deep reinforcement learning, in: International Conference on Machine Learning, 2016.
  • (18) V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, M. Riedmiller, Playing atari with deep reinforcement learning, arXiv preprint arXiv:1312.5602.
  • (19) T. de Bruin, J. Kober, K. Tuyls, R. Babuška, The importance of experience replay database composition in deep reinforcement learning, in: Deep Reinforcement Learning Workshop, NIPS, 2015.
  • (20) K. Narasimhan, T. D. Kulkarni, R. Barzilay, Language understanding for text-based games using deep reinforcement learning

    , in: L. Màrquez, C. Callison-Burch, J. Su, D. Pighin, Y. Marton (Eds.), Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, The Association for Computational Linguistics, 2015, pp. 1–11.

  • (21) A. White, J. Modayil, R. S. Sutton, Surprise and curiosity for big data robotics, in: AAAI-14 Workshop on Sequential Decision-Making with Big Data, Quebec City, Quebec, Canada, 2014.
  • (22) S. B. Thrun, Efficient exploration in reinforcement learning, Tech. rep. (1992).
  • (23) A. W. Moore, C. G. Atkeson, Prioritized sweeping: Reinforcement learning with less data and less time, Machine learning 13 (1) (1993) 103–130.
  • (24) H. van Seijen, R. S. Sutton, Planning by Prioritized Sweeping with Small Backups, in: Proceedings of the 30th International Conference on Machine Learning, Cycle 3, Vol. 28 of JMLR Proceedings,, 2013, pp. 361–369.
  • (25) H. F. Ólafsdóttir, C. Barry, A. B. Saleem, D. Hassabis, H. J. Spiers, Hippocampal place cells construct reward related sequences through unexplored space, Elife 4 (2015) e06063.
  • (26) L. Buhry, A. H. Azizi, S. Cheng, Reactivation, replay, and preplay: how it might all fit together, Neural plasticity 2011 (2011) 203462.
  • (27)

    A. C. Singer, L. M. Frank, Rewarded outcomes enhance reactivation of experience in the hippocampus, Neuron 64 (6) (2009) 910–921.

  • (28) R. S. Sutton, Integrated architectures for learning, planning, and reacting based on approximating dynamic programming, in: Proceedings of the Seventh Int. Conf. on Machine Learning, 1990, pp. 216–224.
  • (29) M. Chamanbaz, D. Mateo, B. M. Zoss, G. Tokić, E. Wilhelm, R. Bouffanais, D. K. P. Yue, Swarm-enabling technology for multi-robot systems, Front. Robot. AI 4 (2017) Art. 12. doi:10.3389/frobt.2017.00012.
  • (30) F. Fernández, M. Veloso, Building a library of policies through policy reuse, Tech. Rep. CMU-CS-05-174, Computer Science Department, Carnegie Mellon University, Pittsburgh, PA (2005).
  • (31) D. J. Foster, M. A. Wilson, Reverse replay of behavioural sequences in hippocampal place cells during the awake state, Nature 440 (7084) (2006) 680–683.