Humans can naturally learn and perform well at a wide variety of tasks, driven by instinct and practice; more importantly, they are able to justify why they would take a certain action. Artificial agents should be equipped with the same capability, so that their decision making process is interpretable by researchers. Following the enormous success of deep learning in various domains, such as the application of convolutional neural networks (CNNs) to computer vision[23, 22, 24, 36], a need for understanding and analyzing the trained models has arisen. Several such methods have been proposed and work well in this domain, for example for image classification [38, 46, 10], sequential models  or through attention .
Deep reinforcement learning (RL) agents also use CNNs to gain perception and learn policies directly from image sequences. However, little work has been so far done in analyzing RL networks. We found that directly applying common visualization techniques to RL agents often leads to poor results. In this paper, we present a novel technique to generate insightful visualizations for pre-trained agents.
Currently, the generalization capability of an agent is—in the best case—evaluated on a validation set of scenarios. However, this means that this validation set has to be carefully crafted to encompass as many potential failure cases as possible. As an example, consider the case of a self-driving agent, where it is near impossible to exhaustively model all interactions of the agent with other drivers, pedestrians, cyclists, weather conditions, even in simulation. Our goal is to extrapolate from the training scenes to novel states that induce a specified behavior in the agent.
In our work, we learn a generative model of the environment as an input to the agent. This allows us to probe the agent’s behavior in novel states created by an optimization scheme to induce specific actions in the agent. For example we could optimize for states in which the agent sees the only option as being to slam on the brakes; or states in which the agent expects to score exceptionally low. Visualizing such states allows to observe the agent’s interaction with the environment in critical scenarios to understand its shortcomings. Furthermore, it is possible to generate states based on an objective function specified by the user. Lastly, our method does not affect and does not depend on the training of the agent and thus is applicable to a wide variety of reinforcement learning algorithms.
Our contributions are:
This is one of the first works to visualize and analyze deep reinforcement learning agents.
We introduce a series of objectives to quantify different forms of interstingness and danger of states for RL agents.
We evaluate our algorithm on 50 Atari games and a driving simulator, and compare performance across three different reinforcement learning algorithms.
An extensive supplement shows additional comprehensive visualizations on 50 Atari games.
We will describe our method before we will discuss relevant related work from the literature.
We will first introduce the notation and definitions that will be used through out the remainder of the paper. We formulate the reinforcement learning problem as a discounted, infinite horizon Markov decision process, where at every time step the agent finds itself in a state and chooses an action following its policy . Then the environment transitions from state to state given the model . Our goal is to visualize RL agents given a user-defined objective function, without adding constraints on the optimization process of the agent itself, i.e. assuming that we are given a previously trained agent with fixed parameters .
We approach visualization via a generative model over the state space and synthesize states that lead to an interesting, user-specified behavior of the agent. This could be, for instance, states in which the agent expresses high uncertainty regarding which action to take or states in which it sees no good way out. This approach is fundamentally different than saliency-based methods as they always need an input for the test-set on which the saliency maps can be computed. The generative model constrains the optimization of states to induce specific agent behavior.
2.1 State Model
Often in feature visualization for CNNs, an image is optimized starting from random noise. However, we found this formulation too unconstrained, often ending up in local minima or fooling examples (Figure 4a). To constrain the optimization problem we learn a generative model on a set
of states generated by the given agent that is acting in the environment. The model is inspired by variational autoencoders (VAEs) and consists of an encoder
that maps inputs to a Gaussian distribution in latent space and a decoderthat reconstructs the input. The training of our generator has three objectives. First, we want the generated samples to be close to the manifold of valid states . To avoid fooling examples, the samples should also induce correct behavior in the agent and lastly, sampling states needs to be efficient. We encode these goals in three corresponding loss terms.
The role of is to ensure that the reconstruction is close to the input such that is minimized. We observe that in the typical reinforcement learning benchmarks, such as Atari games, small details—e.g. the ball in Pong or Breakout—are often critical for the decision making of the agent. However, a typical VAE model tends to yield blurry samples that are not able to capture such details. To address this issue, we model the reconstruction error with an attentive loss
term, which leverages the saliency of the agent to put focus on critical regions of the reconstruction. The saliency maps are computed by guided backpropagation of the policy’s gradient with respect to the state.
As discussed earlier, gradient based reconstruction methods might not be ideal for explaining a CNN’s reasoning process . Here however, we only use it to focus the reconstruction on salient regions of the agent and do not use it to explain the agent’s behavior for which these methods are ideally suited. This approach puts emphasis on details (salient regions) when training the generative model.
Since we are interested in the actions of the agent on synthesized states, the second objective is used to model the perception of the agent:
where is a generic formulation of the output of the agent. For a DQN for example, , i.e. the final action is the one with the maximal -value. This term encourages the reconstructions to be interpreted by the agent the same way as the original inputs . The last term ensures that the distribution predicted by the encoder
stays close to a Gaussian distribution. This allows us to initialize the optimization with a reasonable random vector later and forms the basis of a regularizer. Thus, after training, the model approximates the distribution of statesby sampling from . We will now use the generator inside an optimization scheme to generate state samples that satisfy a user defined target objective.
2.2 Sampling States of Interest
Training a generator with the objective function of Equation 1 allows us to sample states that are not only visually close to the real ones, but which the agent can also interpret and act upon as if they were states from a real environment.
We can further exploit this property and formulate an energy optimization scheme to generate samples that satisfy a specified objective. The energy operates on the latent space of the generator and is defined as the sum of a target function on agent’s policy and a regularizer
The target function can be defined freely by the user and depends on the agent that is being visualized. For a DQN, one could for example define as the Q-value of a certain action, e.g. pressing the brakes of a car. In section 2.3, we show several examples of targets that are interesting to analyze. The regularizer can again be chosen as the KL divergence between
and the normal distribution:
forcing the samples that are drawn from the distribution to be close to the Gaussian distribution that the generator was trained with. We can optimize Equation 4 with gradient descent on .
2.3 Target Functions
Depending on the agent, one can define several interesting target functions – we present and explore seven below, which we refer to as: , , , , , , and action maximization. For a DQN the previously discussed action maximization is interesting to find situations in which the agent assigns a high value to a certain action e.g. . Other states of interest are those to which the agent assigns a low (or high) value for all possible actions . Consequently, one can optimize towards a low -value for the highest valued action with the following objective:
where controls the sharpness of the soft maximum formulation. Analogously, one can maximize the lowest -value with . We can also optimize for interesting situations in which one action is of very high value and another is of very low value by defining
The energy (Equation 4) can be optimized with gradient descent on .
3 Related Work
We divide prior work into two parts. First we discuss the large body of visualization techniques developed primarily for image recognition, followed by related efforts in reinforcement learning.
3.1 Feature Visualization
In the field of computer vision, there is a growing body of literature on visualizing features and neuron activations of CNNs. As outlined in, we differentiate between saliency methods, that highlight decision-relevant regions given an input image, methods that synthesize an image (pre-image) that fulfills a certain criterion, such as activation maximization  or input reconstruction, and methods that are perturbation-based, i.e. they quantify how input modification affects the output of the model.
3.1.1 Saliency Methods
Saliency methods typically use the gradient of a prediction or neuron at the input image to estimate importance of pixels. Following gradient magnitude heatmaps and class activation mapping , more sophisticated methods such as guided backpropagation [39, 28], excitation backpropagation , GradCAM  and GradCAM++  have been developed.  distinguish between regions in favor and regions speaking against the current prediction.  distinguish between sensitivity and implementation invariance.
3.1.2 Perturbation Methods
Perturbation methods modify a given input to understand the importance of individual image regions.  slide an occluding rectangle across the image and measure the change in the prediction, which results in a heatmap of importance for each occluded region. This technique is revisited by 
who introduce blurring/noise in the image, instead of a rectangular occluder, and iteratively find a minimal perturbation mask that reduces the classifier’s score, while train a network for masking salient regions.
3.1.3 Input Reconstruction
As our method synthesizes inputs to the agent, the most closely related work includes input reconstruction techniques.  reconstruct an image from an average of image patches based on nearest neighbors in feature space.  propose to reconstruct images by inverting representations learned by CNNs, while  train a CNN to reconstruct the input from its encoding.
When maximizing the activation of a specific class or neuron, regularization is crucial because the optimization procedure—starting from a random noise image and maximizing an output—is vastly under-constrained and often tends to generate fooling examples that fall outside the manifold of realistic images . In  total variation (TV) is used for regularization, while  propose an update scheme based on Sobolev gradients. In  Gaussian filters are used to blur the pre-image or the update computed in every iteration. Since there are usually multiple input families that excite a neuron,  propose an optimization scheme for the distillation of these clusters.  show that even CNNs with random weights can be used for regularization. More variations of regularization can be found in [34, 35]. Instead of regularization, [30, 31]
use a denoising autoencoder and optimize in latent space to reconstruct pre-images for image classification.
3.2 Explanations for Reinforcement Learning
In deep reinforcement learning however, feature visualization is to date relatively unexplored.  apply t-SNE  on the last layer of a deep Q-network (DQN) to cluster states of behavior of the agent.  also use t-SNE embeddings for visualization, while  examine how the current state affects the policy in a vision-based approach using saliency methods.  use saliency methods from  to visualize the value and advantage function of their dueling Q-network.  finds critical states of an agent based on the entropy of the output of a policy. Interestingly, we could not find prior work using activation maximization methods for visualization. In our experiments we show that the typical methods fail in the case of RL networks and generate images far outside the manifold of valid game states, even with all typical forms of regularization. In the next section, we will show how to overcome these difficulties.
In this section we thoroughly evaluate and analyze our method on Atari games  using the OpenAI Gym  and a driving simulator. We present qualitative results for three different reinforcement learning algorithms, show examples on how the method helps finding flaws in an agent, analyze the loss contributions and compare to previous techniques.
4.1 Implementation details
In all our experiments we use the same factors to balance the loss terms in Equation 6: for the KL divergence and for the agent perception loss. The generator is trained on frames (using the agent and an -greedy policy with ). Optimization is done with Adam  with a learning rate of and a batch size of for epochs. Training takes approximately four hours on a Titan Xp. Our generator uses a latent space of dimensions, and consists of four encoder stages comprised of a15]
and ReLU layer. The starting number of filters is 32 and is doubled at every stage. A fully connected layer is used for mean and log-variance prediction. Decoding is inversely symmetric to encoding, using deconvolutions and halving the number of channels at each of the four steps.
For the experiments on the Atari games we train a double DQN  for two million steps with a reward discount factor of . The input size is pixels. Therefore, our generator performs up-sampling by factors of , up to a output, which is then center cropped to pixels. The agents are trained on grayscale images, for better visual quality however, our generator is trained with color frames and convert to grayscale using a differentiable, weighted sum of the color channels. In the interest of reproducibility we will make the visualization code available.
4.2 Visualizations On Atari Games
In Figure 1, we show qualitative results from various Atari games using different target functions , as described in Section 2.3. From these images we can validate that the general visualizations that are obtained from the method are of good quality and can be interpreted by a human. generates generally high value states independent of a specific action (first row of Figure 1), while generates low reward situations, such as close before losing the game in Seaquest (Figure 1.e) or when there are no points to score (Figure 1.i). Critical situations can be found by maximizing the difference between lowest and highest estimated Q-value with . In those cases, there is clearly a right and a wrong action to take. In Name This Game (Figure 1.d) this occurs when close to the air refill pipe, which prevents suffocating under water; in Kung Fu Master when there are enemies coming from both sides (Figure 1.g), the order of attack is critical, especially since the health of the agent is low (yellow/blue bar on top). An example of maximizing the value of a single action (similar to maximizing the confidence of a class when visualizing image classification CNNs) can be seen in (Figure 1.f) where the agent sees moving left and avoiding the enemy as the best choice of action.
and train our generative model with the same hyperparameters as above and without any modifications on the agent. We present theobjective for the ACKTR agent in Figure 2 to visualize states with both high and low rewards, for example low oxygen (surviving vs. suffocating) or close proximity to enemies (earning points vs. dying).
Compared to the DQN visualizations the ACKTR visualizations, are almost identical in terms of image quality and interpretability. This supports the notion that our proposed approach is independent of the specific RL algorithm.
4.4 Interpretation of Visualizations
Analyzing the visualizations on Seaquest, we make an interesting observation. When maximizing the -value for the actions, in many samples we see a low or very low oxygen meter. In these cases the submarine would need to ascend to the surface to avoid suffocation. Although the up action is the only sensible choice in this case, we also obtain visualized low oxygen states for all other actions. This implies that the agent has not understood the importance of resurfacing when the oxygen is low. We then run several roll outs of the agent and see that the major cause of death is indeed suffocation and not collision with enemies. This shows the impact of visualization, as we are able to understand a flaw of the agent. Although it would be possible to identify this flawed behavior directly by analyzing the frames of training data for our generator, it is significantly easier to review a handful of samples from our method. Further, as the generator is a generative model, we can synthesize states that are not part of its training set.
4.5 Ablation Studies (Loss Terms)
In this section we analyze the three loss terms of our generative model. The human perceptual loss is weighted by the (guided) gradient magnitude of the agent in Equation 2. In Figure 3 we visualize this mask for a DQN agent for random frames from the dataset. The masks are blurred with an averaging filter of kernel size . We observe that guided backpropagation results in precise saliency maps focusing on player and enemies that then focus the reconstructions on what is important for the agent.
To study the influence of the loss terms we perform an experiment in which we evaluate the agent not on the real frames but on their reconstructions. If the reconstructed frames are perfect, the agent with generator goggles achieves the same score as the original agent. We can use this metric to understand the quantitative influence of the loss terms. In Pong, the ball is the most important visual aspect of the game for decision making.
In Table 1 we see that the VAE baseline scores much lower than our model. This can be explained as follows. Since the ball is very small, it is mostly ignored by the reconstruction loss of a VAE. The contribution of one pixel to the overall loss is negligible and the VAE never focuses on reconstructing the important part of the image. Our formulation is built to regain the original performance of the agent, by reweighing the loss on perceptually salient regions of the agent. Overall, we see that our method always improves over the baseline but does not always match the original performance.
4.6 Comparison with Activation Maximization
For image classification tasks, activation maximization works well when optimizing the pre-image directly [27, 3]. However we find that for reinforcement learning, the features learned by the network are not complex enough to reconstruct meaningful pre-images, even with sophisticated regularization techniques. The pre-image converges to a fooling example maximizing the class but being far away from the manifold of states of the environment.
In Figure 4.a we compare our results with the reconstructions generated using the method of  for a DQN agent. We obtain similarly bad pre-images with TV-regularization , Gaussian blurring  and other regularization tricks such as random jitter, rotations, scaling and cropping . This shows that it is not possible to directly apply common techniques for visualizing RL agents and explains why a learned regularization from our generator is needed to produce meaningful examples.
4.7 Experiments with a Driving Simulator
Driving a car is a continuous control task set within a much more complex environment than Atari games. To explore the behavior of our proposed technique in this setting we have created a 3D driving simulation environment and trained an A2C agent maximizing speed while avoiding pedestrians that are crossing the road.
In our first set of experiments we trained an A2C agent to maximize speed while avoiding swerving and pedestrians that are crossing the road. The input to the agent are eight temporal frames comprised of depth, semantic segmentation and a gray-scale image (Figure 6). With this experiment we visualize three random samples for two target functions. The moving car and person categories, appear most prominently when probing the agent for the break action. However, we are also able to identify a flaw: unnecessary braking on empty roads as shown in the left most image of the right most block of three frames. Inappropriate breaking is a well known issue in this problem domain.
In a second set of experiments, we use our simulator to build two custom environments and validate that we can identify problematic behavior in the agent. The agent is trained with four temporal semantic segmentation frames ( pixels) as input (Figure 7). We train the agent in a “reasonable pedestrians” environment, where pedestrians cross the road carefully, when no car is coming or at traffic lights. With these choices, we model data collected in the real world, where it is unlikely that people unexpectedly run in front of the car. We visualize states in which the agent expects a low future return ( objective) in Figure 7. It shows that the agent is aware of other cars, traffic lights and intersections. However, there are no generated states in which the car is about to collide with a person, meaning that the agent does not recognize the criticality of pedestrians. To verify our suspicion, we test this agent in a “distracted pedestrians” environment where people cross the road looking at their phones without paying attention to approaching cars. We find that the agent does indeed run over humans. With this experiment, we show that our visualization technique can identify biases in the training data just by critically analyzing the sampled frames.
4.8 Novel states
To be able to generate novel states is useful, since it allows the method to model new scenarios that were not accounted for during training of the agent. This allows the user to identify potential problems without the need to include every possible permutation of situations in the simulator or real-world data collection.
While one could simply examine the experience replay buffer to find scenarios of interest, our approach allows unseen scenarios to be synthesized. To quantitatively evaluate the assertion that our generator is capable of generating novel states, we sample states and compare them to their closest frame in the training set under an MSE metric. We count a pixel as different if the relative difference in a channel exceeds 25% and report the histogram in Table 2. The results show that there are very few samples that are very close to the training data. On average a generated state is different in 25% of the pixels, which is high, considering the overall common layout of the road, buildings and sky.
We examine these results visually for Atari SeaQuest in Fig. 5, where we show a generated frame and the L2-closest frame from the training set additional to the closest frame in the training set based on the objective function. Retrieval with L2 is, as usual not very meaningful on images since the actual interesting parts of the images are dominated by the background. Thus we have also included a retrieval experiment based on the objective score which shows the submarine in a similar gameplay situation but with different enemy locations. The results in Tab. 2 and Fig. 5 confirm that the method is able to generate unseen states and does not overfit to the training set.
5 Discussion and Conclusions
We have presented a method to synthesize inputs to deep reinforcement learning agents based on generative modeling of the environment and user-defined objective functions. The agent perception loss helps the reconstructions to focus on regions of the current state that are important to the agent and avoid generating fooling examples. Training the generator to produce states that the agent perceives as those from the real environment enables optimizing its latent space to sample states of interest. Please consult the supplementary material included with this submission for more extensive visualization experiments.
We believe that understanding and visualizing agent behavior in safety critical situations is a crucial step towards creating safer and more robust agents using reinforcement learning. We have found that the methods explored here can indeed help accelerate the detection of problematic situations for a given learned agent. For our car simulation experiments we have focused upon the identification of weaknesses in constructed scenarios; however, we see great potential to apply these techniques to much more complex simulation environments where less obvious safety critical weaknesses may be lurking.
We would like to thank Iro Laina, Alexandre Piché, Simon Ramstedt, Evan Racah and Adrien Ali Taiga for helpful discussions and proofreading. We thank the Open Philanthropy project for supporting C.R. while he was an intern at Mila, where this work began.
-  J. Adebayo, J. Gilmer, I. Goodfellow, and B. Kim. Local explanation methods for deep neural networks lack sensitivity to parameter values. International Conference on Learning Representations, Workshop Track, 2018.
-  J. Adebayo, J. Gilmer, M. Muelly, I. Goodfellow, M. Hardt, and B. Kim. Sanity checks for saliency maps. arXiv preprint arXiv:1810.03292, 2018.
-  M. Baust, F. Ludwig, C. Rupprecht, M. Kohl, and S. Braunewell. Understanding regularization to visualize convolutional neural networks. arXiv preprint arXiv:1805.00071, 2018.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling.
The arcade learning environment: An evaluation platform for general
Journal of Artificial Intelligence Research, 47:253–279, 2013.
-  G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym, 2016.
-  A. Chattopadhay, A. Sarkar, P. Howlader, and V. N. Balasubramanian. Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 839–847. IEEE, 2018.
-  P. Dabkowski and Y. Gal. Real time image saliency for black box classifiers. In Advances in Neural Information Processing Systems, pages 6970–6979, 2017.
-  A. Dosovitskiy and T. Brox. Inverting convolutional networks with convolutional networks. CoRR abs/1506.02753, 2015.
-  D. Erhan, Y. Bengio, A. Courville, and P. Vincent. Visualizing higher-layer features of a deep network. University of Montreal, 1341(3):1, 2009.
-  R. C. Fong and A. Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. arXiv preprint arXiv:1704.03296, 2017.
-  S. Greydanus, A. Koul, J. Dodge, and A. Fern. Visualizing and understanding atari agents. arXiv preprint arXiv:1711.00138, 2017.
-  F. Grün, C. Rupprecht, N. Navab, and F. Tombari. A taxonomy and library for visualizing learned features in convolutional neural networks. arXiv preprint arXiv:1606.07757, 2016.
-  S. Hooker, D. Erhan, P.-J. Kindermans, and B. Kim. Evaluating feature importance estimates. arXiv preprint arXiv:1806.10758, 2018.
-  S. H. Huang, K. Bhatia, P. Abbeel, and A. D. Dragan. Establishing appropriate trust via critical states. International Conference on Intelligent Robots (IROS), 2018.
S. Ioffe and C. Szegedy.
Batch normalization: Accelerating deep network training by reducing
internal covariate shift.
International Conference on Machine Learning, pages 448–456, 2015.
-  A. Karpathy, J. Johnson, and L. Fei-Fei. Visualizing and understanding recurrent networks. International Conference on Learning Representations, 2016.
-  P.-J. Kindermans, S. Hooker, J. Adebayo, M. Alber, K. T. Schütt, S. Dähne, D. Erhan, and B. Kim. The (un) reliability of saliency methods. arXiv preprint arXiv:1711.00867, 2017.
-  P.-J. Kindermans, K. T. Schütt, M. Alber, K.-R. Müller, and S. Dähne. Patternnet and patternlrp–improving the interpretability of neural networks. arXiv preprint arXiv:1705.05598, 2017.
-  D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 2015.
-  D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
-  I. Kostrikov. Pytorch implementations of reinforcement learning algorithms. https://github.com/ikostrikov/pytorch-a2c-ppo-acktr, 2018.
-  A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105, 2012.
-  Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
J. Long, E. Shelhamer, and T. Darrell.
Fully convolutional networks for semantic segmentation.
Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015.
-  J. L. Long, N. Zhang, and T. Darrell. Do convnets learn correspondence? In Advances in Neural Information Processing Systems, pages 1601–1609, 2014.
-  L. v. d. Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579–2605, 2008.
-  A. Mahendran and A. Vedaldi. Understanding deep image representations by inverting them. In Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on, pages 5188–5196. IEEE, 2015.
-  A. Mahendran and A. Vedaldi. Visualizing deep convolutional neural networks using natural pre-images. International Journal of Computer Vision, 120(3):233–255, 2016.
-  V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning, pages 1928–1937, 2016.
-  A. Nguyen, A. Dosovitskiy, J. Yosinski, T. Brox, and J. Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In Advances in Neural Information Processing Systems, pages 3387–3395, 2016.
-  A. Nguyen, J. Yosinski, Y. Bengio, A. Dosovitskiy, and J. Clune. Plug & play generative networks: Conditional iterative generation of images in latent space. arXiv preprint arXiv:1612.00005, 2016.
-  A. Nguyen, J. Yosinski, and J. Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 427–436, 2015.
-  A. Nguyen, J. Yosinski, and J. Clune. Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks. arXiv preprint arXiv:1602.03616, 2016.
-  C. Olah, A. Mordvintsev, and L. Schubert. Feature visualization. Distill, 2017. https://distill.pub/2017/feature-visualization.
-  C. Olah, A. Satyanarayan, I. Johnson, S. Carter, L. Schubert, K. Ye, and A. Mordvintsev. The building blocks of interpretability. Distill, 2018. https://distill.pub/2018/building-blocks.
-  S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pages 91–99, 2015.
-  R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. See https://arxiv. org/abs/1610.02391 v3, 7(8), 2016.
-  K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
-  J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806, 2014.
-  M. Sundararajan, A. Taly, and Q. Yan. Axiomatic attribution for deep networks. arXiv preprint arXiv:1703.01365, 2017.
-  D. Ulyanov, A. Vedaldi, and V. Lempitsky. Deep image prior. arXiv preprint arXiv:1711.10925, 2017.
-  Z. Wang, T. Schaul, M. Hessel, H. Van Hasselt, M. Lanctot, and N. De Freitas. Dueling network architectures for deep reinforcement learning. In Proceedings of the 33rd International Conference on International Conference on Machine Learning-Volume 48, pages 1995–2003. JMLR. org, 2016.
-  Y. Wu, E. Mansimov, R. B. Grosse, S. Liao, and J. Ba. Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation. In Advances in neural information processing systems, pages 5285–5294, 2017.
-  K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning, pages 2048–2057, 2015.
-  T. Zahavy, N. Ben-Zrihem, and S. Mannor. Graying the black box: Understanding dqns. In International Conference on Machine Learning, pages 1899–1908, 2016.
-  M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818–833. Springer, 2014.
-  J. Zhang, Z. Lin, J. Brandt, X. Shen, and S. Sclaroff. Top-down neural attention by excitation backprop. In European Conference on Computer Vision, pages 543–559. Springer, 2016.
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba.
Learning deep features for discriminative localization.In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
-  L. M. Zintgraf, T. S. Cohen, T. Adel, and M. Welling. Visualizing deep neural network decisions: Prediction difference analysis. arXiv preprint arXiv:1702.04595, 2017.
To show an unbiased and wide variety of results, in the following, we will show four random samples generated by our method for a DQN agent trained on many of the Atari benchmark environments. We show visualizations optimized for a meaningful objective for each game (e.g. not optimizing for unused buttons). All examples were generated with the same hyperparameter settings.
Please note that for some games better settings can be found. Some generators on visually more complex games would benefit from longer training to generate sharper images. Our method is able to generate reasonable images even when the DQN was unable to learn a meaningful policy such as for Montezuma’s revenge (Fig. 39). We show two additional objectives maximizing/minimizing the expected reward of the state under a random action: and . Results in alphabetical order and best viewed in color.