Artificial neural networks enjoy increasing popularity for image classification tasks. They have shown excellent performance in large scale competitions . One reason is the ability to train neural networks with millions of training samples by parallelizing them on GPU hardware. This allows to use numbers of training samples which match the large number of parameters in deep neural networks. However, understanding what region of the image is important for a classification decision, is still an open question for neural networks, as well as for many other non-linear models. The work of  proposed Layer-wise Relevance Propagation (LRP) as a solution for explaining what pixels of an image are relevant for reaching a classification decision. This was done for neural networks, bag of word models [3, 10], and in a subsequent work 
, for Fisher vectors.
This paper proposes an approach to extend LRP to neural networks with nonlinearities beyond the commonly used neural network formulation. One example of such nonlinearities are local renormalization layers which can not be handled by standard LRP 
. The presented approach is based on first (or higher) order Taylor expansion. We consider a classification setup with real-valued outputs. A classifieris a mapping of an input space such that denotes the presence of the class.
2 Layer-wise Relevance Propagation for Neural Networks
In the following we consider neural networks consisting of layers of neurons. The outputof a neuron
is a non-linear activation functionas given by
Given an image and a classifier the aim of layer-wise relevance propagation is to assign each pixel of a pixel-wise relevance score such that
Pixels with contain evidence against the presence of a class, while is considered as evidence for the presence of a class. These pixel-wise relevance scores can be visualized as an image called heatmap (see Fig. 1 for examples). Obviously, many possible such decompositions exist which satisfy equation 2. The work of  yield pixel-wise decompositions which are consistent with evaluation measures  and human intuition.
Assume that we know the relevance of a neuron at network layer for the classification decision , then we like to decompose this relevance into messages sent to those neurons at the layer which provide inputs to neuron such that equation 3 holds.
We can then define the relevance of a neuron at layer by summing all messages from neurons at layer as in equation 4
The work in  established two formulas for computing the messages . The first formula called -rule is given by
with and . The variable is a “stabilizer” term whose purpose is to avoid numerical degenerations when is close to zero, and which is chosen to be small. The second formula called -rule is given by
where the positive and negative weighted activations are treated separately. The variable controls how much inhibition is incorporated in the relevance redistribution. A fairly large value for (e.g. ) leads to sharper heatmaps. In both formulas the message has the following structure
The meaningfulness of the resulting pixel-wise decomposition for the input layer comes from the fact that the terms are derived from the weighted activations
of the input neurons. Note that layer-wise relevance propagation does not use gradients in contrast to backpropagation during the training phase. For full details on layer-wise relevance propagation the reader is referred to.
3 Extending LRP to local renormalization layers
We consider a general neuron whose pooling and activation does not fit into the structure given by equation 1, and consequently, intuition for a possible redistribution formula is lacking. In this paper we propose a strategy for such neurons, based on the Taylor expansion of its activation function. A Taylor-based approach was used in 
for decomposing ReLU neurons by exploiting their local linearity. Here, we consider instead fully nonlinear neurons.
Suppose we can define for each neuron input to neuron a term which is derived from its activation such that . Then we can define a message . Such messages were used in equations 5 and 6 where the weighting was chosen to depend on the weighted activations of neuron : and , respectively. For differentiable neurons, such weighting can be obtained by performing a first order Taylor expansion. Let be a nonlinear activation function. Then, by Taylor expansion at some reference point , we get
Elements of the sum can be assigned to incoming neurons, and the zero-order term can be redistributed equally between them, leading to the decomposition
of the neuron activation onto its input neurons. Local renormalization layers have been shown to improve the performance in deep neural networks . Consider the local renormalization of a neuron by the set of its surrounding neurons as
This interaction can be modeled by a layer in the network that has an activation function as given in equation 10. Local renormalization layers represent a non-linearity which cannot be tackled exactly by LRP as introduced in , however the strategy proposed above can be applied.
One choice to be made is the point at which to perform the Taylor expansion. There are two apparent candidates, firstly the actual input to the renormalization layer and, secondly, the input corresponding to the case when only the neuron fires which is to be normalized . The partial derivative of at is zero for all variables with due to
This implies that the Taylor approximation has no off-diagonal contribution.
Therefore we apply the Taylor series around the point :
This weighting satisfies the following qualitative properties: for the neuron input which is to be normalized, the sign of the relevance is kept. For suppressing neighboring neurons , , the sign of the relevance can be flipped in line with their suppressing property. The absolute value of the relevance received by the suppressing neurons is proportional to the square of their input. In the limits and , the local renormalization converges against the identity, and the approximation recovers the identity. A baseline to compare against is to treat the normalization as constant. In that case the weights for the relevance propagation in equation 3 become a zero one vector, the relevance is propagated only to that neuron which is to be normalized: if and only if is the neuron which is to be normalized by neuron .
We need to define a measure for meaningfulness and quality of a pixel-wise decomposition in order to evaluate the various strategies to compute it. Here we use an idea from : A pixel is considered highly relevant for the classification score of the image if modifying it by assigning it a random RGB value , and classifying the modified image results in a strong decrease of the real-valued classification score . This idea can be extended by sequentially modifying pixels from the most relevant to the least relevant. The result is a graph of the prediction score as a function of the number of modified pixels. An example for some sequences which will be explained below is shown in Fig. 2. We can use these graphs to evaluate the meaningfulness of a pixel-wise decomposition.
In the first experiment we compare the measure when flipping highest-scoring pixels first, against flipping pixels in random order, and against flipping lowest scoring pixels first. If the classifier is able to identify pixels that are important for classification, then flipping highest scoring pixels first should result in the fastest decaying curve, while flipping lowest scoring pixels first should result in the slowest decrease. Fig. 2 tests this property on the CIFAR-10 dataset  which consists of 50000 images of size drawn from 10 object classes. Scores are averaged over the 5000 images of the test set of CIFAR-10 for a classifier in which local renormalization layers are treated as the identity during computation of pixel-wise scores. Experiments corroborate that flipping highest scoring pixels first results in the fastest decrease of the prediction score on average over the test set. The decrease is sharper compared to random flipping, or flipping lowest scoring pixels first.
In a second experiment we compare which treatment of the local renormalization layer is best to identify those pixels that are most relevant for classifying an image. The two tested approaches for treating the local renormalization are (1) like it would be the identity, (2) by first order Taylor expansion as given by equation 15. These approaches are furthermore tested when used in conjunction with the two methods proposed by , namely, the -rule in equation 5 with a fixed value of the numerical stabilizer , and the -rule shown in equation 6, with fixed .
|rule for basic layers||rule for normalization layers||AUC score|
|eq. 4,5,||first-order Taylor||35.47|
|eq. 4,6,||first-order Taylor||53.82|
We measure the quality of heatmaps by perturbing highest pixels first and computing the area under the curve (AUC). Lower AUC averaged over a large number of images indicates a better identification of pixel relevance by the heatmap. Results on CIFAR-10 are shown in Table 1. We observe that in all cases using first order Taylor in normalization layers improves the heatmap AUC score. This shows its effectiveness for dealing with non-linear neuron layers.
We perform the same experiments also with Imagenet  and MIT Places  datasets, each time evaluating results for images from their respective unlabeled test sets. Note that computing a heatmap requires only a predicted class label, not a ground truth. We evaluated results for the parameter settings , in equation 6 and , , in equation 5. Table 2 shows the difference of AUC between variants of LRP, when using either the identity or the Taylor expansion for local renormalization layers. We observe the following ordering starting with the lowest (best) AUC: , , , , . This order holds independent of whether we consider Imagenet or MIT places, when using Taylor for local renormalization layers. When using identity instead of Taylor, the order remain the same, except for and that are swapped. This is by itself an interesting result demonstrating that use of Taylor in the normalization layer does not disrupt the overall properties of relevance propagation techniques. For a comparison to other approaches such as heatmaps based on deconvolutions , or backpropagated gradients  we refer to .
Table 3 shows the difference of AUC between Taylor and identity for local renormalization layers, for various choices of datasets and LRP parameters. We observe that for the parameters with best AUC ( and ), using Taylor expansion for representing local renormalization layers further improves the AUC scores. For the remaining choices the results are on par or slightly worse. This is consistent with the interpretation of large values of as smoothing out small contributions. It is also consistent with the observation that and yield both smooth heatmaps in general. Heatmaps for some parameters of interest are shown in Figure 3. Taylor with has both high pixel selectivity and low noise, which in agreement with its measured superiority in the quantitative experiments.
We have presented an extension of layer-wise relevance propagation (LRP) based on first-order Taylor expansions for product-type nonlinearities. Such nonlinearities occur in the local renormalization layers of deep convolutional neural networks. The proposed extension is evaluated on three popular datasets and it is shown to clearly outperform the original LRP method. In future work we will investigate the potential gain of using higher order Taylor expansions, and apply the method to a larger class of neural network layers.
-  Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE, 10(7):e0130140, 2015.
-  Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. Analyzing classifiers: Fisher vectors and deep neural networks. In Proc. IEEE CVPR, 2016.
Gabriela Csurka, Christopher R. Dance, Lixin Fan, Jutta Willamowski, and
Visual categorization with bags of keypoints.
In Workshop on Statistical Learning in Computer Vision, ECCV, pages 1–22, 2004.
-  Alex Krizhevsky. Learning multiple layers of features from tiny images. http://www.cs.toronto.edu/ kriz/cifar.html, 2009.
-  Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1106–1114, 2012.
-  Grégoire Montavon, Sebastian Bach, Alexander Binder, Wojciech Samek, and Klaus-Robert Müller. Explaining nonlinear classification decisions with deep taylor decomposition. arXiv:1512.02479, 2015.
-  Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, pages 1–42, April 2015.
-  Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Bach, and Klaus-Robert Müller. Evaluating the visualization of what a deep neural network has learned. CoRR, abs/1509.06321, 2015.
-  Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. CoRR, abs/1312.6034, 2013.
Koen E. A. van de Sande, Theo Gevers, and Cees G. M. Snoek.
Evaluating color descriptors for object and scene recognition.IEEE Trans. Pattern Anal. Mach. Intell., 32(9):1582–1596, 2010.
-  Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, pages 818–833, 2014.
Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva.
Learning deep features for scene recognition using places database.In Adv. in NIPS, pages 487–495, 2014.