1 Introduction
As deep neural networks have gained popularity in many scientific disciplines and technological applications, there is a growing interest in understanding the representations they learn and the computations they perform. One approach towards achieving such understanding is to visualize the features that activate the neurons in a network. There is a growing body of work that seeks to visualize features by synthesizing images which maximally drive hidden layer units. While this approach can give us a rough intuition about a unit’s selectivity, it provides only a very incomplete picture of its computation. In addition to characterizing feature detectors by the stimulus that elicits the largest response, it is important to identify the nuisance parameters to which the neuron is invariant. As hidden layers build up response invariances gradually with depth, it is not the
image that most strongly drives a unit that is the most telling about this unit’s function, but instead the set of images that elicit a strong response. While some previous work has visualized multiple ‘facets’ of neurons’ selectivity, these efforts focused mostly on the highest layers of the network and relied on initialization or random sampling strategies to create multiple images for each unit. However, as we show in the present paper, these approaches underestimate the true diversity of the selectivity of even relatively lowlevel units. Additionally, these approaches have not offered insights about how the representations of different networks trained on the same task compare. Our contributions are the following:
Motivated by the phase invariance of complex cells in the early visual system of the brain, we show why visualizing invariance is as important as visualizing selectivity for understanding the computations of even lowlevel units.

We develop a nonparametric approach to map the manifold of highlyactivating inputs as exhaustively as possible.

We show that even relatively lowlevel units exhibit a remarkable degree of invariance in VGG19 [28], which is not revealed by finding highly activating stimuli from multiple optimization runs with random initializations.

We find that in low to intermediate layers of VGG19, at least two types of invariances emerge: tolerance to local diffeomorphic transformations tuned to specific features, and phase invariance, where units respond well to periodic texture patterns and are insensitive to their phase. We additionally offer a way to quantify these invariances.

In contrast, we find that low to intermediate layers of a network with skip connections (ResNet50 [11]) that was trained on the same task as VGG19 exhibit far less phase invariance, revealing representational differences between these two networks.

We showcase our visualization approach on a CNN trained to predict responses to natural images in primary visual cortex of the primate brain.
We provide the code to replicate our results. ^{1}^{1}1https://github.com/sacadena/diverse_feature_vis
2 Related work
One way to identify selectivity of hidden units is to look for image patches in the dataset that drive them maximally [6, 33]. These image patches can sometimes hint at a unit’s selectivity, but it can be difficult to identify their common features. Optimizationbased techniques have proven more useful for feature visualization: a common approach is to search for preimages that drive individual neurons maximally via gradient ascent [6]. Most previous work focused on deep layers, where finding naturallooking preimages is challenging. For example, the activation objective leads to adversariallike patterns [20, 29]. As a consequence, much of the followup work focused on developing regularization techniques to obtain more natural preimages, including penalties on highfrequency noise [16, 20] or the distance between the generated visualizations and natural images patches [32], or performing gradient descent in the feature space of a deep generator network [19].
Goodfellow et al. [9] were the first (to our knowledge) to study invariances in deep networks. Their approach allows to quantify how invariant a unit is to known transformations such as translation, (3D) rotation or scaling, but it does not allow to discover these transformations if they are unknown in advance.
Recent work proposes visualizing multiple ‘facets’ of the neuron’s selectivity by obtaining multiple images from different random initializations [17], using a diverse set of highly activating images as initializations [21], or using a generative image model to sample highlyactivating images [18].
These methods do not explicitly specify an objective to produce a diverse set of images. In contrast, we optimize a batch of images to drive the neuron of interest strongly while simultaneously being as distinct from each other as possible. Recent concurrent work [22]
introduces a similar idea, albeit with a different loss function based on texture representations
[7, 8].3 Discovering invariances
3.1 Motivation: simple and complex cells
We illustrate our point by considering a toy example well known from early vision in the brain (Fig. 1): simple and complex cells [12]
, which are found in the primary visual cortex, an early stage of visual processing in the mammalian brain. Simple cells can be approximated well by a linear filter followed by a thresholding nonlinearity (e.g. ReLU). The linear filter usually resembles a Gabor filter. Complex cells are, like simple cells, selective for a specific orientation and spatial frequency. However, unlike simple cells they respond to Gabor patches of arbitrary phases – they are phaseinvariant. The standard model for this phase invariance is the socalled energy model (Fig.
1A, [1]), which sums over the squared responses of two Gabor filters phaseshifted by 90 (Fig. 1C, black). This energy model has also been used to study rotation, scaling and more general invariances in the context of unsupervised representation learning [2, 3, 15]An alternative formulation was originally proposed by Hubel & Wiesel, who discovered complex cells in the 1960ies in the primary visual cortex of cats [12]. Their model suggests that complex cells are the result of pooling over multiple simple cells with a range of phase preferences (Fig. 1B). If the learned weights and phase preferences exhibit some variability, the resulting phase invariance is only approximate (Fig. 1C, blue).
Now, consider what happens when we study simple and complex cells using activity maximization. For a simple cell, we will recover its selectivity. For a complex cell, however, all Gabor patches of optimal orientation and spatial frequency will elicit a high response, irrespective of their phase. In the case of the Energy Model, which is perfectly phaseinvariant, we may obtain this set of optimal images by starting with random initializations. However, for an imperfect model more likely to occur in reality (e. g. Hubel & Wiesel model, blue in Fig. 1C), there is a unique maximum, which we will find despite the fact that activations are consistently above 80% of the maximum for all phases. Thus, activity maximization will produce the same result for both simple and complex cells (a single Gabor patch), but this result will miss the key aspect of the complex cell’s computation: its phase invariance.
3.2 Mapping invariances
3.2.1 Objective.
The idea behind our approach is to find a batch of images in which each image maximally drives a specific unit while the images are maximally different from one another. Starting with a batch of images
, initialized as white noise, we
maximize the following objective using gradient ascent:(1) 
Here, is the output activation of unit in layer for the image in the batch, is the likelihood of the image under a generative model of natural images and is a distance between two images, The likelihood and distance measures are specified below. Note that we set the image size to the receptive field size of units in the layer to be visualized, such that the outputs are
spatially and we can omit the indices over space. We constrain the norm of the synthesized images to be equal to half the average norm of natural images patches of the same size taken from the ImageNet dataset
^{2}^{2}2Using half the average norm is a heuristic that we use because the synthesized images tend to be localized to the center of the patch.
, where we assume that zero in each color channel corresponds to the average value of this channel across the ImageNet training set. For visualization, we add this mean and clip the values between 0 and 255. Very few pixels fall outside this range.The first and the second term in the objective are similar to previous work, encouraging the optimization to find natural images that strongly activate the unit. The third term forces all images in the batch to be as distinct as possible from all other images, since we penalize the minimum distance between any pair of images. This objective presents a tradeoff: we allow for some degree of nonmaximal responses if this allows us to increase the set of strongly activating preimages substantially.
It is important to use the minimum distance in the objective rather than the average. Maximizing the average distance does not necessarily lead to coverage of the invariant subspace. Consider the Energy Model: assuming we generate an even number of images, the optimal solution maximizing the average distance is to place all images at either of two distinct phases separated by . Now we fail to generate a diverse set of images but the average distance is high (). In contrast, the desired solution of images evenly separated by will give a smaller average distance for and can be obtained when maximizing the minimum distance.
It has also some advantages to consider a single unit within a feature map compared to considering the entire feature map. When maximizing the activation of the entire feature map, the resulting image will be shiftinvariant by construction and properties such as phase invariance of individual units cannot be detected.
3.2.2 Natural image prior.
We use PixelCNN++ [27] as a natural image prior, as it allows directly evaluating and optimizing the likelihood of an image patch of arbitrary size. In a nutshell, PixelCNN++ improves upon PixelCNN [23]
and earlier autoregressive models
[24, 30, 31]that attempt to capture the distribution of natural images by expressing the joint distribution of all pixels as the product of the distributions of individual pixels conditioned on a causal neighborhood. We use the model pretrained on Cifar100 provided by OpenAI
^{3}^{3}3https://github.com/openai/pixelcnn which is stateoftheart in terms of likelihood on natural images.3.2.3 Distance metric.
To evaluate the distance between two images, we use a feature space given by the neural network to encourage diversity on perceptually interesting image properties. For an output unit in layer , we compute the Euclidean distance in the feature space of the preceding convolutional layer:
(2) 
where and
are vectors of activations in the preceding layer flattened over space and channels.
3.2.4 Optimization.
We optimize the objective defined in Eq. (1) using the Adam optimizer [13] with a learning rate of 0.1 until the objective converges (maximum of 1000 steps). Similar to Olah et al. [22], we precondition the gradient to reduce the effect of high frequencies by dividing each frequency component by .
We manually set the hyperparameter
, which controls the strength of the natural image prior, based on qualitative inspection of the resulting images in an exploratory experiment. We used for all experiments.We sweep a range of values for (0.02, 0.04, 0.08, … 20.48) and for each unit pick the largest such
that the average activation level remains above a threshold. This threshold is 80% of the maximum for the complex cell model and 90% for VGG19 and ResNet50. See Fig.
2A and Fig. 5 for a qualitative justification of these thresholds.3.3 Application to complex cell models
Before applying our approach to a deep neural network, we verify that it works when the units are only approximately invariant to some transformation. To this end, we use the Hubel & Wiesel model of a complex cell outlined above (Fig. 1B), which does not produce perfect phase invariance, but still responds strongly to Gabor patches of all phases.
Indeed, our approach can visualize the entire invariant subspace spanning the full range of phases (Fig. 2). Without the diversity term (), the optimization tends to converge to the same preimage (Fig. 2B). Four out of six solutions correspond to the globally most strongly driving image (see also Fig. 1C, top). In contrast, with an appropriate choice of , the images distribute uniformly (Fig. 2B, C). If we increase too much, however, the diversity penalty becomes too large and the optimization will converge to solutions including nonoptimal images. Thus, to visualize the invariant subspace, we should pick the largest that leads to only a small decrease in activation level. This point depends on how ‘clean’ the invariance of the cell is. For the Hubel & Wiesel model considered here, this drop in activation occurs when the average activation falls below 80% of the maximum, which corresponds to the response range for images within the approximately invariant subspace (see Fig. 1C, blue line).
Note that for the simple cell, which does not exhibit any such response invariance, the curve looks qualitatively different (Fig. 2A, red line). Thus, we can quantify response invariance of units in a DNN by computing the minimum distance between any two images in the batch at the optimal .
4 Invariances in VGG19
We asked to what extent deep neural networks trained on largescale object recognition (ImageNet [25]) exhibit response invariances in their convolutional layers. Previous work focused mostly on higher layers and did not find much invariance in low and intermediate layers. However, in neuroscience it is wellknown that low and midlevel neurons in the brain – like complex cells – can exhibit a substantial degree of response invariance. Moreover, there is evidence for a considerable degree of similarity between neural representations in DNNs trained on object recognition and the primate visual system [14, 10, 4, 5]. In particular, we have shown [4] that the convolutional layers of VGG19 [28] around layer conv3_1 best predict neural activity in primary visual cortex, including that of many complex cells. Therefore we would expect that these layers in the VGG19 network should also exhibit some degree of invariance to phase and potentially other transformations.
4.1 Convolutional layers of VGG19 exhibit response invariances
We start by considering two example units from layer conv3_2 (Fig. 3) of VGG19. As in the complex cell example, we can increase the diversity of generated images quite substantially while maintaining a high activation level (Fig. 3A). Only when we increase too much, the activation level drops substantially and the images start deteriorating (Fig. 3B, top row). Overall, the tradeoff between image diversity and activation level looks qualitatively similar to the complex cell example above.
Moreover, the images generated with the optimal look significantly more diverse than those obtained by random initialization at (Fig. 3B, middle and bottom rows). Indeed, most units showed quite some degree of invariance: we can increase the image diversity considerably while maintaining activation levels above 90% of the maximum (Fig. 5 for conv3_1; see Sect. 1 in the Supp. for additional convolutional layers). Below, we therefore use the largest such that maintains the average activation level above 90% of the maximum.
4.2 Response invariances are a learned property of the network
Is this invariance a learned property of the network or does it arise trivially from the network architecture? We repeated the analysis on a network with the same architecture as VGG19 but random weights. To keep the two networks comparable, we normalized both the activations and the distances between images such that they are equal to one for . We found that units in the random network are substantially less invariant than those of VGG19 (Fig. 5A), suggesting that the neurons’ response invariance is indeed a learned property. Remarkably, by introducing the diversity term into the preimage search, we could increase the minimum distance between any two images in a batch by a factor of at least two and up to 100fold without ‘sacrificing’ more than 10% of the unit’s activation level (Fig. 5B), a property that the random network does not exhibit.
4.3 Types of invariance: texture vs. shape detectors
We now investigate the types of invariance learned by different units in the network. We start by considering two example units from layer conv3_1 (Fig. 6). The first unit responds to a dark grid on brighter background of arbitrary color. In addition to this selectivity, it appears to be entirely phase and rotationinvariant: the location of the grid lines and their orientation is irrelevant for the unit’s activation, but their general spatial scale and the foreground color are important. We refer to units that exhibit this property as texture detectors.
The second unit, in contrast, detects a circular feature in the lower half of its receptive field. While it is sensitive to the location of this pattern within its receptive field, it exhibits a substantial degree of color and scale invariance: the contours have a sinusoidal crosssection whose local phase varies across images, such that by using linear combinations of multiple of these images one can obtain the circular pattern in various different sizes and color combinations. We refer to such units as shape detectors: they are sensitive to location but allow for some degree of local diffeomorphic transformation.
The two units shown here are representative of a larger number of units in various layers of VGG19 (see Fig. 7 and Sect. 2 from Supp. for more examples). As we will quantitatively show below, they lie on two extremes of a spectrum along which we can characterize low and intermediatelevel units.
4.4 Quantification of phase invariance (textures)
So far, we have described texture and shape units only qualitatively. We therefore developed metrics to quantify these properties more systematically. We start by quantifying phase invariance, the property that characterizes texture detectors.
While shift equivariance is built into CNNs, phase invariance of individual units has to our knowledge not been reported. A perfectly phaseinvariant unit would maintain a high activation when presented with shifted versions of its preferred texture. Therefore, to quantify phase invariance, we optimize an image twice as large as the unit’s receptive field such that the average activation of all possible windowed crops from this image is maximized (Fig. 8A, 1–4). Indeed, for a decent number of units we had qualitatively labeled as ‘texture detectors,’ the crops generated in this way (Fig. 8A, 3) resemble the templates we synthesized earlier (Fig. 8A, 4) and elicit similarly high activations (Fig. 8C). On the other hand, ‘shapeselective’ units expect certain structures in specific locations within their receptive field. Generating a texture where arbitrary crops are highly activating is not possible for these units (Fig. 8B).
To quantify this intuitive argument, we defined shift invariance as the ratio between the average activation of all crops from the larger texture and the average activation of the diverse templates produced earlier (see example histogram in Fig. 8C, for conv3_1). Indeed, the units labeled as phaseinvariant (Fig. 8A), maintain a high activations despite arbitrary phase shifts, while the activation of the shapeselective units (Fig. 8B) drops substantially (Fig. 8C).
Note that synthesizing a larger image by maximizing all crops is similar to maximizing an entire channel’s activity (i. e. feature map) for a sufficiently large input image, an approach other authors have taken for feature visualization [22]. Although insightful in many occasions, the drawback is that this procedure often occludes shape selectivity. For instance, the first unit in Fig. 8B is selective to a circular pattern in the topright with rays pointing towards the bottomleft when maximized individually. However, the resulting texture looks like a field of oriented edges, thus missing the crucial pattern that drives this unit.
4.5 Tolerance to local deformations (shapes)
The second invariance we identify is tolerance to local deformations. A closer look at some examples (e. g. Fig 6, right; Fig. 8B, top) reveals that some of the units have local tolerance for phase changes. The patterns these units are tuned for can be locally built by spatially arranging multiple complexcelllike quadrature pairs. This would suggest, that – although mapped into a nonlinear feature space – linear combinations of the ‘template’ images spanning the invariant subspace should highly activate these units as well. We illustrate this seemingly counterintuitive hypothesis with a toy example and then show how it applies to CNNs.
Consider the following example comprised of two complex cells arranged such that they detect a topleft corner (Fig. 9). The unit allows for individually shifting up or down the horizontal edge, and left or right the vertical edge. Each of the two edges is detected by an energy model of a complex cell (Fig. 9
A), each at a defined location within the receptive field. Accordingly, the highly activating template images are made up of combinations of odd and even Gabors (Fig.
9B) and any linear combination of them is again a highly activating image (Fig. 9C).To quantify whether the same property holds for VGG units, we computed the average activation level of linear combinations of the maximally activating images. Specifically, we took the averages (in pixel space) of all 15 pairs of templates (Fig. 8A.5), renormalized them to the same norm as the templates and compared their average activation to that of the templates. For ‘textureselective’ units this procedure deteriorates the clear texture patterns revealed by the templates (see for instance Fig. 8A.5). Accordingly, the unit’s activation level to these images drops substantially (Fig. 8D, red+orange). We quantify this drop by computing a linear combination index, defined as the ratio between the average activation of averageimage pairs and the average activation of the diverse templates. Units tuned to shape patterns that are tolerant to local transformations give averagepairs that are fairly similar to the original templates, producing a high linear combination index.
4.6 Characterization of invariances across layers
We have identified two metrics that quantify two different forms of invariance in VGG units. Our examples from Fig. 8 suggest that these two types of invariance are anticorrelated. As this does not have to be the case a priori – a complex cell would score high on both metrics – we asked whether this was just due to our selection of examples or whether it holds more generally across layers. Indeed, shift invariance and tolerance to local deformations appear to be anticorrelated across a wide range of layers (Fig. 8E; conv3 in particular). We also observe that higher layers tend to be less shiftinvariant than lower ones (e. g. compare within conv3 in Fig. 8E).
5 Diverse visualizations of early layers of ResNet50
To test whether our results so far are properties of VGG19 or apply more generally to CNNs trained on ImageNet, we also applied our methods to ResNet50 [11]. We considered its early layers up to conv3_1 (fourth block), which have receptive field sizes comparable to the layers we studied in VGG19. We first synthesized diverse image batches with different diversity penalties and found a similar tradeoff between activation and diversity as found before (see Sect. 3 in Suppl. Material). However, for the that evoked at least 90% of the maximal responses we observed on average a smaller diversity compared to that of VGG19 units. We then ran our analysis to identify both phase and shape invariance and surprisingly found a much reduced number of phaseinvariant units compared to VGG19 (Fig. 10): there are basically no ResNet50 units for which the crops from the optimal texture look like the optimized templates (e.g. Fig. 10A,2+3). On the other hand, templatepair averages do not appear to qualitatively deviate from the synthesized templates (Fig. 10A,4) indicating a strong presence of tolerance to local changes. The two metrics introduced above confirm this observation quantitatively: the distribution of shift invariance indices is shifted towards zero in ResNet50 layers (Fig. 10B) with respect to those in VGG19.
This is a very interesting finding, because it shows that the different architectures learn quite different features in their early layers despite both being trained on ImageNet and achieving comparable classification accuracy. Thus, our novel approach to feature visualization helped us identify strong representational differences in the canonical directions between two architectures that would not have been observed with conventional activity maximization
6 Phase invariance in Primary visual cortex (V1)
As a final practical use case, we applied our method to a threelayer CNN that has been trained to predict neural responses in V1 when monkeys are shown natural images (data from [4]; see also their Fig. 3). Our method unveils the known cell types – simple: phaseselective and complex: phaseinvariant (Fig 11). Although complex cells can also be identified using specifically designed stimuli or analysis methods relying on quadratic features (e. g. spiketriggered covariance [26]), our nonparametric approach could in principle also uncover other types of invariance that are not captured by quadratic features. Given that we see no such additional invariances, there are likely no other major features V1 cells are invariant to – a conclusion that could not be drawn using parametric approaches.
7 Conclusion
Motivated by early vision in the brain, we investigated the response invariances in the early to intermediate convolutional layers of DNNs. We found that units in early layers of VGG19 show invariance to global texturepreserving transformations and invariance to local shapepreserving transformations. In contrast, ResNet50 does not exhibit the same degree of shift invariance. This difference could explain why practitioners working on texture synthesis and style transfer observe that the features of VGG work substantially better than those of more modern architectures such as residual networks.
We conclude that these methods not only give new insights into the computations performed by DNNs and how they compare with other architectures, but also constitutes an important step towards a unified language for describing neural representations in both biological and computer vision.
Acknowledgements.
We thank Jonas Rauber and Andreas Tolias for useful discussions. This work was supported by the German Research Foundation (DFG) grant EC 479/11 to A.S.E. The International Max Planck Research School for Intelligent Systems (IMPRSIS) supported S.A.C. The work was also supported by IARPA via Department of Interior (DoI) contract D16PC00003.
References
 [1] Adelson, E.H., Bergen, J.R.: Spatiotemporal energy models for the perception of motion. J. Opt. Soc. Am. A 2(2), 284–299 (1985). https://doi.org/10.1364/JOSAA.2.000284
 [2] Berkes, P., Wiskott, L.: Slow feature analysis yields a rich repertoire of complex cell properties. Journal of vision 5(6), 9–9 (2005)

[3]
Bethge, M., Gerwinn, S., Macke, J.H.: Unsupervised learning of a steerable basis for invariant image representations. In: Human Vision and Electronic Imaging XII. vol. 6492, p. 64920C. International Society for Optics and Photonics (2007)
 [4] Cadena, S.A., Denfield, G.H., Walker, E.Y., Gatys, L.A., Tolias, A.S., Bethge, M., Ecker, A.S.: Deep convolutional models improve predictions of macaque v1 responses to natural images. bioRxiv (2017). https://doi.org/10.1101/201764
 [5] Cadieu, C.F., Hong, H., Yamins, D.L., Pinto, N., Ardila, D., Solomon, E.A., Majaj, N.J., DiCarlo, J.J.: Deep neural networks rival the representation of primate IT cortex for core visual object recognition. PLoS computational biology 10(12), e1003963 (2014), 00152
 [6] Erhan, D., Bengio, Y., Courville, A., Vincent, P.: Visualizing higherlayer features of a deep network. Tech. Rep. 1341, University of Montreal (Jun 2009), also presented at the ICML 2009 Workshop on Learning Feature Hierarchies, Montréal, Canada.

[7]
Gatys, L., Ecker, A.S., Bethge, M.: Texture synthesis using convolutional neural networks. In: Advances in Neural Information Processing Systems. pp. 262–270 (2015)

[8]
Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2414–2423 (2016)
 [9] Goodfellow, I., Lee, H., Le, Q.V., Saxe, A., Ng, A.Y.: Measuring invariances in deep networks. In: Advances in neural information processing systems. pp. 646–654 (2009)
 [10] Güçlü, U., van Gerven, M.A.J.: Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. Journal of Neuroscience 35(27), 10005–10014 (2015). https://doi.org/10.1523/JNEUROSCI.502314.2015
 [11] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
 [12] Hubel, D.H., Wiesel, T.N.: Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. The Journal of physiology 160(1), 106 (1962), 09139
 [13] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
 [14] Kriegeskorte, N.: Deep neural networks: A new framework for modeling biological vision and brain information processing. Annual Review of Vision Science 1(1), 417–446 (2015). https://doi.org/10.1146/annurevvision082114035447
 [15] Lies, J.P., Häfner, R.M., Bethge, M.: Slowness and sparseness have diverging effects on complex cell learning. PLoS computational biology 10(3), e1003468 (2014)
 [16] Mahendran, A., Vedaldi, A.: Understanding deep image representations by inverting them. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5188–5196 (2015)
 [17] Mahendran, A., Vedaldi, A.: Visualizing deep convolutional neural networks using natural preimages. International Journal of Computer Vision 120(3), 233–255 (2016)
 [18] Nguyen, A., Clune, J., Bengio, Y., Dosovitskiy, A., Yosinski, J.: Plug & play generative networks: Conditional iterative generation of images in latent space. In: CVPR. vol. 2, p. 7 (2017)
 [19] Nguyen, A., Dosovitskiy, A., Yosinski, J., Brox, T., Clune, J.: Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In: Advances in Neural Information Processing Systems. pp. 3387–3395 (2016)
 [20] Nguyen, A., Yosinski, J., Clune, J.: Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In: The IEEE Conference on Computer Vision and Pattern Recognition (June 2015)

[21]
Nguyen, A.M., Yosinski, J., Clune, J.: Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks. Visualization for Deep Learning workshop, ICML (2016)
 [22] Olah, C., Mordvintsev, A., Schubert, L.: Feature visualization. Distill (2017). https://doi.org/10.23915/distill.00007
 [23] van den Oord, A., Kalchbrenner, N., Espeholt, L., Vinyals, O., Graves, A., et al.: Conditional image generation with pixelcnn decoders. In: Advances in Neural Information Processing Systems. pp. 4790–4798 (2016)
 [24] Oord, A.v.d., Kalchbrenner, N., Kavukcuoglu, K.: Pixel recurrent neural networks. arXiv preprint arXiv:1601.06759 (2016)
 [25] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., FeiFei, L.: ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) 115(3), 211–252 (2015). https://doi.org/10.1007/s112630150816y
 [26] Rust, N.C., Schwartz, O., Movshon, J.A., Simoncelli, E.P.: Spatiotemporal elements of macaque v1 receptive fields. Neuron 46(6), 945–956 (2005)
 [27] Salimans, T., Karpathy, A., Chen, X., Kingma, D.P., Bulatov, Y.: Pixelcnn++: A pixelcnn implementation with discretized logistic mixture likelihood and other modifications. In: Submitted to ICLR 2017 (2016)
 [28] Simonyan, K., Zisserman, A.: Very deep convolutional networks for largescale image recognition. arXiv preprint arXiv:1409.1556 (2014), http://arxiv.org/abs/1409.1556
 [29] Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., Fergus, R.: Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 (2013)
 [30] Theis, L., Bethge, M.: Generative image modeling using spatial lstms. In: Advances in Neural Information Processing Systems. pp. 1927–1935 (2015)
 [31] Theis, L., Hosseini, R., Bethge, M.: Mixtures of conditional gaussian scale mixtures applied to multiscale image representations. PloS one 7(7), e39857 (2012)
 [32] Wei, D., Zhou, B., Torrabla, A., Freeman, W.: Understanding intraclass knowledge inside cnn. arXiv preprint arXiv:1507.02379 (2015)
 [33] Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: European conference on computer vision. pp. 818–833. Springer (2014)
8 Supplementary
8.1 Diversity/activation maximization tradeoff VGG19
As in Figures 4 and 5, we show here the tradeoff between diversity and activity maximization for all layers including the natural image prior. Diversity is measured as the minimum distance in feature space between all pairs of synthesized templates. Each curve represents a unit (feature map) of the corresponding layer. The curves connect the average of three optimization runs for a choice of from Equation 1. The penalty for the natural image prior was set to after visual inspection. The curves were normalized to the maximum sum of activations (relative average activation level). On the left: The tradeoff between minimum distance and relative average activation. On the right: The same curves normalized to have a unit minimum distance. This facilitated comparison with the network with random weights (black). Here, we show in black a sample of units from a random network with the same architecture as VGG19. Note that the VGG units exhibit more invariance at each layer than expected from random weights for all studied layers.
8.1.1 conv1_2
.
conv2_1
.
conv2_2
.
conv3_1
.
conv3_2
.
conv3_3
.
conv3_4
.
8.2 Example invariant subspaces at optimal for early convolutional layers of VGG19
conv1_2
.
conv2_1
.
conv2_2
.
conv3_1
.
conv3_2
.
conv3_3
.
conv3_4
.
8.3 Diversity/activation maximization tradeoff ResNet50
conv2_1
.
conv2_2
.
conv2_3
.
conv3_1
.
8.4 Example invariant subspaces at optimal for early convolutional layers of ResNet50
conv2_1
.
conv2_2
.
conv2_3
.
conv3_1
.