1 Introduction
Many learning problems involve inferring properties of temporally extended sequences given an objective function. For instance, in reinforcement learning (RL), the task is to find a policy that maximizes expected future discounted rewards (value). RL algorithms fall into two main classes: (1) modelfree algorithms that learn cached value functions directly from sample trajectories, and (2) modelbased algorithms that estimate transition and reward functions, from which values can be computed using treesearch or dynamic programming. However, there is a third class, based on the
successor representation(SR), that factors the value function into a predictive representation and a reward function. Specifically, the value function at a state can be expressed as the dot product between the vector of expected discounted future state occupancies and the immediate reward in each of those successor states.
Representing the value function using the SR has several appealing properties. It combines computational efficiency comparable to modelfree algorithms with some of the flexibility of modelbased algorithms. In particular, the SR can adapt quickly to changes in distal reward, unlike modelfree algorithms. In this paper, we also highlight a feature of the SR that has been less wellinvestigated: the ability to extract bottleneck states (candidate subgoals) from the successor representation under a random policy stachenfeld2014design
. These subgoals can then be used within a hierarchical RL framework. In this paper we develop a powerful function approximation algorithm and architecture for the SR using a deep neural network, which we call
Deep Successor Reinforcement Learning (DSR). This enables learning the SR and reward function from raw sensory observations with endtoend training.The DSR consists of two subcomponents: (1) a reward feature learning component, constructed as a deep neural network, predicts intrinsic and extrinsic rewards to learn useful features from raw observations; and (2) an SR component, constructed as a separate deep neural network, that estimates the expected future “feature occupancy” conditioned on the current state and averaged over all actions. The value function can then be estimated as the dot product between these two factored representations. We train DSR by sampling experience trajectories (state, nextstate, action and reward) from an experience replay memory and apply stochastic gradient descent to optimize model parameters. To avoid instability in the learning algorithm, we interleave training of the successor and reward components.
We show the efficacy of our approach on two different domains: (1) learning to solve goals in gridworld domains using the MazeBase game engine and (2) learning to navigate a 3D maze to gather a resource using the Doom game engine. We show the empirical convergence results on several policy learning problems as well as sensitivity of the value estimator given distal reward changes. We also demonstrate the possibility of extracting plausible subgoals for hierarchical RL by performing normalizedcuts on the SR shi2000normalized .
2 Related work
The SR has been used in neuroscience as a model for describing different cognitive phenomena. gershman2012successor showed that the temporal context model howard2002distributed , a model of episodic memory, is in fact estimating the SR using the temporal difference algorithm. corneil2015attractor introduced a model based on SR for preplay and rapid path planning in the CA3 region of the hippocampus. They interpret the SR as an an attractor network in a low–dimensional space and show that if the network is stimulated with a goal location it can generate a path to the goal. stachenfeld2014design suggested a model for tying the problems of navigation and reward maximization in the brain. They claimed that the brain’s spatial representations are designed to support the reward maximization problem (RL); they showed the behavior of the place cells and grid cells can be explained by finding the optimal spatial representation that can support RL. Based on their model they proposed a way for identifying reasonable subgoals from the spectral features of the SR. Other work (see for instance, botvinick2014model ; daw2014algorithmic ) have also discussed utilizing the SR for subgoal and option discovery.
There are also models similar to the SR that have been been applied to other RLrelated domains. schraudolph1994temporal introduced a model for evaluating the positions in the game of Go; the model is reminiscent of SR as it predicts the fate of every position of the board instead of the overall game score. Another rewardindependent model, universal option model (UOM), proposed in szepesvari2014universal , uses state occupancy function to build a general model of options. They proved that UOM of an option, given a reward function, can construct a traditional option model. There has also been a lot of work on option discovery in the tabular setting mcgovern2001automatic ; csimcsek2005identifying ; mannor2004dynamic ; menache2002q ; konidaris2009skill . In more recent work, Machado et al. machado2016learning presented an option discovery algorithm where the agent is encouraged to explore regions that were previously out of reach. However, option discovery where nonlinear state approximations are required is still an open problem.
Our model is also related to the literature on value function approximation using deep neural networks. The deepQ learning model mnih2015human and its variants (e.g., silver2016mastering ; schaul2015prioritized ; nair2015massively ; mnih2016asynchronous ) have been successful in learning Qvalue functions from highdimensional complex input states.
3 Model
3.1 Background
Consider an MDP with a set of states , set of actions , reward function , discount factor , and a transition distribution . Given a policy , the Qvalue function for selecting action in state is defined as the expected future discounted return:
(1) 
where, is the state visited at time and the expectation is with respect to the policy and transition distribution. The agent’s goal is to find the optimal policy which follows the Bellman equation:
(2) 
3.2 The successor representation
The SR can be used for calculating the Qvalue function as follows. Given a state , action and future states , SR is defined as the expected discounted future state occupancy:
where when its argument is true and zero otherwise. This implicitly captures the state visitation count. Similar to the Bellman equation for the Qvalue function (Eq. 2), we can express the SR in a recursive form:
(3) 
Given the SR, the Qvalue for selecting action in state can be expressed as the inner product of the immediate reward and the SR dayan1993improving :
(4) 
3.3 Deep successor representation
For large state spaces, representing and learning the SR can become intractable; hence, we appeal to nonlinear function approximation. We represent each state by a dimensional feature vector which is the output of a deep neural network parameterized by .
For a feature vector , we define a featurebased SR as the expected future occupancy of the features and denote it by . We approximate by another deep neural network parameterized by : . We also approximate the immediate reward for state as a linear function of the feature vector : , where is a weight vector. Since reward values can be sparse, we can also train an intrinsic reward predictor . A good intrinsic reward channel should give dense feedback signal and provide features that preserve latent factors of variations in the data (e.g. deep generative models that do reconstruction). Putting these two pieces together, the Qvalue function can be approximated as (see 4 for closed form):
(5) 
The SR for the optimal policy in the nonlinear function approximation case can then be obtained from the following Bellman equation:
(6) 
where .
3.4 Learning
The parameters can be learned online through stochastic gradient descent.
The loss function for
is given by:where and the parameter denotes a previously cached parameter value, set periodically to . This is essential for stable Qlearning with function approximations (see mnih2015human ).
For learning , the weights for the reward approximation function, we use the following squared loss function:
(7) 
Parameter is used for obtaining the , the shared feature representation for both reward prediction and SR approximation. An ideal should be: 1) a good predictor for the immediate reward for that state and 2) a good discriminator for the states. The first condition can be handled by minimizing loss function ; however, we also need a loss function to help in the second condition. To this end, we use a deep convolutional autoencoder to reconstruct images under an L2 loss function. This dense feedback signal can be interpreted as an intrinsic reward function. The loss function can be stated as:
(8) 
The composite loss function is the sum of the three loss functions given above:
(9) 
For optimizing Eq. 9, with respect to the parameters , we iteratively update and . That is, we learn a feature representation by minimizing ; then given , we find the optimal . This iteration is important to ensure that the successor branch does not backpropagate gradients to affect . We use experience replay memory of size to store transitions, and apply stochastic gradient descent with a learning rate of , momentum of , a discount factor of and the exploration parameter annealed from 1 to 0.1 as training progresses. Algorithm 1 highlights the learning algorithm in greater detail.
4 Automatic Subgoal Extraction
Learning policies given sparse or delayed rewards is a significant challenge for current reinforcement learning algorithms. This is mainly due to inefficient exploration schemes such as greedy. Existing methods like Boltzmann exploration and Thomson sampling stadie2015incentivizing ; osband2016deep offer significant improvements over greedy, but are limited due to the underlying models functioning at the level of basic actions. Hierarchical reinforcement learning algorithms barto2003recent such as the options framework szepesvari2014universal ; sutton1999between provide a flexible framework to create temporal abstractions, which will enable exploration at different timescales. The agent will learn options to reach the subgoals which can be used for intrinsic motivation. In the context of hierarchical RL, goel2003subgoal discuss a framework for subgoal extraction using the structural aspects of a learned policy model. Inspired by previous work in subgoal discovery from state trajectories csimcsek2005identifying and the tabular SR stachenfeld2014design , we use the learned SR to generate plausible subgoal candidates.
Given a random policy (), we train the DSR until convergence and collect the SR for a large number of states . Following csimcsek2005identifying ; shi2000normalized
, we generate an affinity matrix
given, by applying a radial basis function (with Euclidean distance metric) for each pairwise entry
in (to generate ). Let be a diagonal matrix with . Then as per shi2000normalized, the second largest eigenvalue of the matrix
gives an approximation of the minimum normalized cut value of the partition of . The states that lie on the endpoints of the cut are plausible subgoal candidates, as they provide a path between a community of state groups. Given randomly sampled from , we can collect statistics of how many times a particular state lies along the cut. We pick the topk states as the subgoals. Our experiments indicate that it is possible to extract useful subgoals from the DSR.5 Experiments
In this section, we demonstrate the properties of our approach on MazeBase sukhbaatar2015mazebase , a gridworld environment, and the Doom game engine kempka2016vizdoom . In both environments, observations are presented as raw pixels to the agent. In the first experiment we show that our approach is comparable to DQN in two goalreaching tasks. Next, we investigate the effect of modifying the distal reward on the initial Qvalue. Finally, using normalizedcuts, we identify subgoals given the successor representations in the two environments.
5.1 Goaldirected Behavior
Solving a maze in MazeBase
We learn the optimal policy in the maze shown in Figure 2 using the DSR and compare its performance to the DQN mnih2015human . The cost of living or moving over water blocks is 0.5 and the reward value is 1. For this experiment, we set the discount rate to 0.99 and the learning rate to . We anneal the from 1 to 0.1 over 20k steps; furthermore, for training the reward branch, we anneal the number of samples that we use, from 4000 to 1 by a factor of 0.5 after each training episode. For all experiments, we prioritize the reward training by keeping a database of nonzero rewards and sampling randomly from the replay buffer with a 0.8 probability and 0.2 from the database. Figure 3 shows the average trajectory (over 5 runs) of the rewards obtained over 100k episodes. As the plot suggests, DSR performs on par with DQN.
Finding a goal in a 3D environment
We created a map with 4 rooms using the ViZDoom platform kempka2016vizdoom . The map is shown in Figure 2. We share the same network architecture as in the case of MazeBase. The agent is spawned inside a room, and can explore any of the other three rooms. The agent gets a perstep penalty of 0.01 and a positive reward of 1.0 after collecting an item from one of the room (highlighted in red in Figure2). As shown in Figure3, the agent is able to successfully navigate the environment to obtain the reward, and is competitive with DQN.
5.2 Value function sensitivity to distal reward changes
The decomposition of value function into SR and immediate reward prediction allows DSR to rapidly adapt to changes in the reward function. In order to probe this, we performed experiments to measure the adaptability of the value function to distal reward changes. Given the gridworld map in Figure2, we can train the agent to solve the goal specified in the map as highlighted in section 5.1. Without changing the goal location, we can change the reward scalar value upon reaching the goal from 1.0 to 3.0. Our hypothesis is that due to the SRbased value decomposition, our value estimate will converge to this change by just updating the reward weights (SR remains same). As shown in Figure 4, we confirm that the DSR is able to quickly adapt to the new value function by just updating .
5.3 Extracting subgoals from the DSR
Following section 4, we can also extract subgoals from the SR. We collect by running a random policy on both MazeBase and VizDoom. During learning, we only update SR () and the reconstruction branch (), as the immediate reward at any state is zero (due to random policy).
As shown in Figures 5 and 6, our subgoal extraction scheme is able to capture useful subgoals and clusters the environment into reasonable segments. Such a scheme can be ran periodically within a hierarchical reinforcement learning framework to aid exploration. One inherent limitation of this approach is that due to the random policy, the subgoal candidates are often quite noisy. Future work should address this limitation and provide statistically robust ways to extract plausible candidates. Additionally, the subgoal extraction algorithm should be nonparametric to handle flexible number of subgoals.
6 Conclusion
We presented the DSR, a novel deep reinforcement learning framework to learn goaldirected behavior given raw sensory observations. The DSR estimates the value function by taking the inner product between the SR and immediate reward predictions. This factorization of the value function gives rise to several appealing properties over existing deep reinforcement learning methods—namely increased sensitivity of the value function to distal reward changes and the possibility of extracting subgoals from the SR under a random policy.
For future work, we plan to combine the DSR with hierarchical reinforcement learning. Learning goaldirected behavior with sparse rewards is a fundamental challenge for existing reinforcement learning algorithms. The DSR can enable efficient exploration by periodically extracting subgoals, learning policies to satisfy these intrinsic goals (skills), and subsequently learning hierarchical policy over these subgoals in an options framework szepesvari2014universal ; kulkarni2016hierarchical ; schaul2015universal . One of the major issues with the DSR is learning discriminative features. In order to scale up our approach to more expressive environments, it will be crucial to combine various deep generative and selfsupervised models eslami2016attend ; greff2015binding ; rezende2016one ; kulkarni2015deep ; whitney2016understanding ; gregor2015draw ; huang2015efficient ; noroozi2016unsupervised with our approach. In addition to subgoals, using DSR for extracting other intrinsic motivation measures such as improvements to the predictive world model schmidhuber2010formal or mutual information mohamed2015variational is worth pursuing.
References
 [1] Andrew G Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete Event Dynamic Systems, 13(4):341–379, 2003.
 [2] Matthew Botvinick and Ari Weinstein. Modelbased hierarchical reinforcement learning and human action control. Philosophical Transactions of the Royal Society of London B: Biological Sciences, 369(1655):20130480, 2014.
 [3] Dane S Corneil and Wulfram Gerstner. Attractor network dynamics enable preplay and rapid path planning in maze–like environments. In Advances in Neural Information Processing Systems, pages 1675–1683, 2015.
 [4] Nathaniel D Daw and Peter Dayan. The algorithmic anatomy of modelbased evaluation. Philosophical Transactions of the Royal Society of London B: Biological Sciences, 369(1655):20130478, 2014.
 [5] Peter Dayan. Improving generalization for temporal difference learning: The successor representation. Neural Computation, 5(4):613–624, 1993.
 [6] SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, Koray Kavukcuoglu, and Geoffrey E Hinton. Attend, infer, repeat: Fast scene understanding with generative models. arXiv preprint arXiv:1603.08575, 2016.
 [7] Samuel J Gershman, Christopher D Moore, Michael T Todd, Kenneth A Norman, and Per B Sederberg. The successor representation and temporal context. Neural Computation, 24(6):1553–1568, 2012.
 [8] Sandeep Goel and Manfred Huber. Subgoal discovery for hierarchical reinforcement learning using learned policies. In FLAIRS conference, pages 346–350, 2003.
 [9] Klaus Greff, Rupesh Kumar Srivastava, and Jürgen Schmidhuber. Binding via reconstruction clustering. arXiv preprint arXiv:1511.06418, 2015.
 [10] Karol Gregor, Ivo Danihelka, Alex Graves, and Daan Wierstra. Draw: A recurrent neural network for image generation. arXiv preprint arXiv:1502.04623, 2015.

[11]
Marc W Howard and Michael J Kahana.
A distributed representation of temporal context.
Journal of Mathematical Psychology, 46(3):269–299, 2002.  [12] Jonathan Huang and Kevin Murphy. Efficient inference in occlusionaware generative models of images. arXiv preprint arXiv:1511.06362, 2015.
 [13] Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaśkowski. Vizdoom: A doombased ai research platform for visual reinforcement learning. arXiv preprint arXiv:1605.02097, 2016.
 [14] George Konidaris and Andre S Barreto. Skill discovery in continuous reinforcement learning domains using skill chaining. In Advances in Neural Information Processing Systems, pages 1015–1023, 2009.
 [15] Tejas D Kulkarni, Karthik R Narasimhan, Ardavan Saeedi, and Joshua B Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. arXiv preprint arXiv:1604.06057, 2016.
 [16] Tejas D Kulkarni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convolutional inverse graphics network. In Advances in Neural Information Processing Systems, pages 2530–2538, 2015.
 [17] Marlos C Machado and Michael Bowling. Learning purposeful behaviour in the absence of rewards. arXiv preprint arXiv:1605.07700, 2016.

[18]
Shie Mannor, Ishai Menache, Amit Hoze, and Uri Klein.
Dynamic abstraction in reinforcement learning via clustering.
In
Proceedings of the twentyfirst international conference on Machine learning
, page 71. ACM, 2004.  [19] Amy McGovern and Andrew G Barto. Automatic discovery of subgoals in reinforcement learning using diverse density. Computer Science Department Faculty Publication Series, page 8, 2001.
 [20] Ishai Menache, Shie Mannor, and Nahum Shimkin. Qcut—dynamic discovery of subgoals in reinforcement learning. In Machine Learning: ECML 2002, pages 295–306. Springer, 2002.
 [21] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. arXiv preprint arXiv:1602.01783, 2016.
 [22] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Humanlevel control through deep reinforcement learning. Nature, 518(7540):529–533, 2015.
 [23] Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. In Advances in Neural Information Processing Systems, pages 2116–2124, 2015.
 [24] Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro De Maria, Vedavyas Panneershelvam, Mustafa Suleyman, Charles Beattie, Stig Petersen, et al. Massively parallel methods for deep reinforcement learning. arXiv preprint arXiv:1507.04296, 2015.
 [25] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. arXiv preprint arXiv:1603.09246, 2016.
 [26] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. arXiv preprint arXiv:1602.04621, 2016.
 [27] Danilo Jimenez Rezende, Shakir Mohamed, Ivo Danihelka, Karol Gregor, and Daan Wierstra. Oneshot generalization in deep generative models. arXiv preprint arXiv:1603.05106, 2016.
 [28] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In Proceedings of the 32nd International Conference on Machine Learning (ICML15), pages 1312–1320, 2015.
 [29] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05952, 2015.
 [30] Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990–2010). Autonomous Mental Development, IEEE Transactions on, 2(3):230–247, 2010.
 [31] Nicol N Schraudolph, Peter Dayan, and Terrence J Sejnowski. Temporal difference learning of position evaluation in the game of go. Advances in Neural Information Processing Systems, pages 817–817, 1994.
 [32] Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 22(8):888–905, 2000.
 [33] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484–489, 2016.
 [34] Özgür Şimşek, Alicia P Wolfe, and Andrew G Barto. Identifying useful subgoals in reinforcement learning by local graph partitioning. In Proceedings of the 22nd international conference on Machine learning, pages 816–823. ACM, 2005.
 [35] Kimberly L Stachenfeld, Matthew Botvinick, and Samuel J Gershman. Design principles of the hippocampal cognitive map. In Advances in neural information processing systems, pages 2528–2536, 2014.
 [36] Bradly C Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint arXiv:1507.00814, 2015.
 [37] Sainbayar Sukhbaatar, Arthur Szlam, Gabriel Synnaeve, Soumith Chintala, and Rob Fergus. Mazebase: A sandbox for learning from games. arXiv preprint arXiv:1511.07401, 2015.
 [38] Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semimdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1):181–211, 1999.
 [39] Csaba Szepesvari, Richard S Sutton, Joseph Modayil, Shalabh Bhatnagar, et al. Universal option models. In Advances in Neural Information Processing Systems, pages 990–998, 2014.
 [40] William F Whitney, Michael Chang, Tejas Kulkarni, and Joshua B Tenenbaum. Understanding visual concepts with continuation learning. arXiv preprint arXiv:1602.06822, 2016.