Any vision system, biological or artificial, has its own biases. These biases emanate from different sources. Two common sources include a) the environment and the data on which the system has been trained, and b) system constraints (e.g., hypothesis class, model parameters). Exploring these biases is important from at least two perspectives. First, it allows to better understand a system (e.g., explain and interpret its decisions). Second, it helps reveal system vulnerabilities and make it more robust against adversarial perturbations and attacks.
In this paper, we recruit two popular methods from computational neuroscience to study the inherent biases in deep neural networks. The first one, called classification images technique, was introduced into visual psychophysics by Ahumada Jr (1996) as a new experimental tool. It has been used to examine visual processing and to understand vision across a variety of tasks including simple detection tasks, visual search, and object recognition. It has also been applied to the auditory domain. See Murray (2011) for a review of the topic. The second method, known as spike triggered analysis (Marmarelis, 2012), is often used to discover the best stimulus to which a neuron responds (e.g., oriented bars). These methods are appealing for our purpose since a) they are general and can be applied to study any black box system (so long it emits a response to an input stimulus) and b) make a modest number of assumptions. From a system identification point of view, they provide a first-order approximation of a complex system such as the brain or an artificial neural network.
By feeding white noise stimuli to a classifier and averaging the ones that are categorized into a particular class, we obtain an estimate of the templates it uses for classification. Unlike classification images experiments in human psychophysics, where running a large number of trials is impractical, artificial systems can often be tested against a large number of inputs. While still a constraint, we will discuss how such problems can be mitigated (e.g., by generating stimuli containing faint structures). Over four datasets, MNIST(LeCun et al., 1998), Fashion-MNIST (Xiao et al., 2017), CIFAR-10 (Krizhevsky et al., 2009), and ImageNet Deng et al. (2009)
, we employ classification images to discover implicit biases of a network, utilize those biases to influence network decisions, and detect adversarial perturbations. We also show how spike triggered averaging can be used to identify and visualize filters in different layers of a CNN. Finally, in a less directly related analysis to classification images, we demonstrate how decisions of a CNN are influenced by varying the signal to noise ratio (akin to microstimulation experiments in monkey electrophysiology or priming experiments in psychophysics). We find that CNNs behave in a similar fashion to their biological counterparts and their responses can be characterized by a psychometric function. This may give insights regarding top-down attention and feedback mechanisms in CNNs (SeeBorji & Itti (2012)).
2 Related works and concepts
Our work relates to a large body of research attempting to understand, visualize, and interpret deep neural networks. These networks have been able to achieve impressive performance on a variety of challenging vision and learning tasks (e.g., Krizhevsky et al. (2012); He et al. (2016)). However, they are still not well understood, have started to saturate in performance (Recht et al., 2019), are brittle222Current deep neural networks can be easily fooled by subtle image alterations in ways that are imperceptible to humans; a.k.a adversarial examples (Szegedy et al., 2013; Goodfellow et al., 2014b)., and continue to trail humans in accuracy and generalization. This calls for a tighter confluence between machine learning, computer vision, and neuroscience. In this regard, the proposed tools here are complementary to the existing ones in the deep learning toolbox.
Perhaps, the closest work to ours is Vondrick et al. (2015) where they attempted to learn biases in the human visual system and transfer those biases into object recognition systems. Some other works (e.g., Fong et al. (2018)) have also used human data (e.g., fMRI, cell recording) to improve the accuracy of classifiers, but have not utilized classification images. Bashivan et al. (2019) used is activation maximization to iteratively change the pixel values in the direction of the gradient to maximize the firing rate of V4 neurons333See https://openreview.net/forum?id=H1ebhnEYDH for a discussion on this.. Unlike these works, here we strive to inspect the biases in classifiers, in particular, neural networks, to improve their interpretability and robustness.
2.1 Classification images
In a typical binary classification image experiment, on each trial, a signal and a noise image are summed to produce the stimulus . The observer is supposed to decide which of the two categories the stimulus belongs to. Classification image is then calculated as:
where is the average of noise patterns in a stimulus–response class of trials. For example, is the average of the noise patterns over all trials where the stimulus contained signal 1 but the observer responded 2. is an approximation of the template that the observer uses to discriminate between the two stimulus classes. The intuition behind the classification images is that the noise patterns in some trials have features similar to one of the signals, thus biasing the observer to choose that signal. By computing the average over many trials a pattern may emerge. can also be interpreted as the correlation map between stimulus and response:
is the pixel-wise standard deviation of the noisen and is the standard deviation of response . High positive correlations occur at spatial locations that strongly influence the observer’s responses. Conversely, very low (close to zero) correlations occur at locations that have no influence on the observer’s responses. Assuming zero-mean noise and an unbiased observer, Eq. 2 reduces to , where is the average of the noise patterns over all trials where the observer gave a response (See Murray (2011) for details). Thus, is the average of the noise patterns over all trials where the observer responded , minus the average over all trials where the observer responded , regardless of which signal was presented.
We have illustrated the classification images concept in Fig. 1 with a binary classifier trained to separate two digits. The stimulus is a linear combination of noise plus signal as follows:
The computed templates for different values444We use classification images, bias map, template, and average noise pattern, interchangeably. Please do not confuse this bias with the bias terms in neural networks., using about 10 million trials, highlight regions that are correlated with one of the digits (here 1 vs. 7 or 5 vs. 6). The template fades away with increasing noise (e.g., ) but it still resembles the template in the low-noise condition (i.e., ).
2.2 Spike triggered analysis
The spike-triggered analysis, also known as “reverse correlation” or “white-noise analysis”, is a tool for characterizing the response properties of a neuron using the spikes emitted in response to a time-varying stimulus. It includes two methods: spike-triggered averaging (STA) and spike-triggered covariance (STC). They provide an estimate of a neuron’s linear receptive field and are useful techniques for the analysis of electrophysiological data. In the visual system, these methods have been used to characterize retinal ganglion cells (Meister et al., 1994; Sakai & Naka, 1987), lateral geniculate neurons (Reid & Alonso, 1995), and simple cells in the primary visual cortex (DeAngelis et al., 1993; Jones & Palmer, 1987). See Schwartz et al. (2006) for a review.
STA is the average stimulus preceding a spike. It provides an unbiased estimate of a neuron’s receptive field only if the stimulus distribution is spherically symmetric (e.g., Gaussian white noise). STC can be used to identify a multi-dimensional feature space in which a neuron computes its response. It identifies the stimulus features affecting a neuron’s response via an eigen-decomposition of the spike-triggered covariance matrix(Sandler & Marmarelis, 2015; Park & Pillow, 2011).
denote a spatio-temporal stimulus vector affecting a neuron’s scalar spike responsein a single time bin. The main goal of neural characterization is to find , a low-dimensional projection matrix such that captures the neuron’s dependence on the stimulus
. The STA and the STC matrix are the empirical first and second moments of the spike-triggered stimulus-response pairs, respectively. They are defined as:
where is the number of spikes and is the total number of time bins. The traditional spike triggered analysis gives an estimate for the basis consisting of: (1)
, if it is significantly different from zero, and (2) the eigenvectors of
corresponding to those eigenvalues that are significantly different from eigenvalues of the prior stimulus covariance. When a stimulus is not white noise (i.e., is correlated in space or time), whitened STA can be written as:
where is a matrix whose th row is the stimulus vector and denotes a column vector whose th element is . The whitened STA is equivalent to linear least-squares regression of the stimulus against the spike train.
Classification images and spike triggered analysis are related in the sense that both estimate the terms of a Wiener/Volterra expansion in which the mapping from the stimuli to the firing rate is described using a low-order polynomial (Marmarelis, 2012). See Dayan et al. (2001) for a discussion on this. Here, we focus on STA and leave STC to future works.
We present four use cases of classification images and STA to examine neural networks, with a focus on CNNs since they are a decent model of human visual processing and are state of the art computer vision models. Our approach, however, is general and can be applied to any classifier. In particular, it is most useful when dealing with black-box methods where choices are limited.
3.1 Understanding and visualizing classifier biases
We trained a CNN with 2 conv layers, 2 pooling layers, and one fully connected layer (see supplement Fig. 10) on the MNIST dataset. This CNN achieves 99.2% test accuracy. We then generated 1 million 28 28 white noise images and fed them to the CNN. The average noise map for each digit class is shown in Fig. 2A. These biases/templates illustrate the regions that are important for classification. Surprisingly, for some digits (0 to 7), it is very easy to tell which digit the bias map represents555Weighting the noise patterns by their classification confidence or only considering the ones with classification confidence above a threshold did not result in significantly different classification images.. We notice that most of the noise patterns are classified as 8, perhaps because this digit has a lot of structure in common with other digits. Feeding the average noise maps back to CNN, they are classified correctly, except 8 which is classified as 2 (see image captions in Fig. 2A).
Classification images of the CNN over MNIST perceptually make sense to humans. This, however, does not necessarily hold across all classifiers and datasets. For example, classification images of a logistic regression classifier on MNIST, shown in Fig. 2B, do not resemble digits (the same happens to MLP and RNN; see supplement Fig. 13). This implies that perhaps CNNs extract features the same way the human visual system does, thus share similar mechanisms and biases with humans. Classification images over the CIFAR-10 dataset, derived using 1 million RGB noise patterns, are shown in Fig. 3A. In contrast to MNIST and Fashion-MNIST (Fig. 5), classification images on CIFAR-10 (using CNNs) do not resemble target classes. One possible reason might be because images are more calibrated and aligned over the former two datasets than CIFAR-10 images.
How much information do the classification images carry? To answer this question, we used bias maps to classify the MNIST test digits. The bias map with the maximum dot product to the test digit determines the output class. The confusion matrix of this classifier is shown in Fig.2C. Using the CNN bias map as a classifier leads to 25.8% test accuracy. The corresponding number for a classifier made of logistic regression bias is 47.6%. Both of these numbers are significantly above 10% chance accuracy. To get an idea regarding the significance of these numbers, we repeated the same using the mean images and logistic regression weights. These two classifiers lead to 63.1% and 83.8% test accuracy, respectively, which are better than the above-mentioned results using bias maps but demand access to the ground-truth data and labels. Over CIFAR-10, classification using bias maps leads to 23.71% test accuracy which is well above chance. Using the mean training images of CIFAR-10 leads to 28.69% test accuracy (Fig. 3B).
Analysis of sample complexity. To get an idea regarding the sample complexity of the classification images approach, we ran three analyses. In the first one, we varied the number of noise patterns as . We found that with 10K noise stimuli, the bias maps already start to look like the target digits (see Fig. 4, and supplement Fig. 14). In the second analysis, we followed Greene et al. (2014)
to generate noise patterns containing subtle structures. Over MNIST and Fashion-MNIST datasets, we used ridge regression to reconstruct all 60K training images from a set of 960 Gabor wavelets (See Appendix for details). We then projected the learned weights (a matrix of size 60K
) to a lower-dimensional space using principal component analysis (PCA). We kept 250 components that explained 96.1% of the variance. To generate a noise pattern, we randomly generated a vector of 250 numbers and projected it back to theD space, using them as weights for image-shaped Gabor wavelets and then sum them to noise image. Over CIFAR-10, we used 1520 Gabor filters for each RGB channel and kept 600 principal components that explained 97.5% of the variance. Classification images using 1M samples generated this way for MNIST, Fashion-MNIST, and CIFAR-10 datasets are shown in Fig. 5
. Classification images resemble the target classes even better now (compared to using white noise). Using the new bias maps for classification, we are able to classify MNIST, Fashion-MNIST and CIFAR-10 test data with 35.5%, 41.21%, and 21.67% accuracy, respectively. In the third analysis, we trained an autoencoder and a variational autoencoder(Kingma & Welling, 2013)
over MNIST, only for 2 epochs. We did so to make the encoders powerful just enough to produce images that contain subtle digit structures (See Fig.15 in supplement). As expected, now the classification images can be computed with much less number of stimuli (100). Results from these analyses suggest that it is possible to lower the sample complexity when some (unlabeled) data is available. This is, in particular, appealing for practical applications of classification images.
Results on ImageNet. We conducted an experiment on ImageNet validation set including 50K images covering 1000 categories and 1 million samples using Gabor PCA sampling (from the above CIFAR-10 experiment over CIFAR-10 images) and pretrained CNNs (on ImageNet train set). As results in Table 1 show, even with 1M samples and without parameter tuning, we obtain an improvement over the chance level (0.0010 or 0.1%. We obtain about 2x accuracy than chance using ResNet152 He et al. (2016). It seems that 1M samples is not enough to cover all classes since no noise pattern is classified under almost half of the classes using ResNet152. For some backbones, even a larger number of classes remain empty (yet another evidence that white noise can reveal biases in models). We believe it is possible to improve these results with more samples. As you can see with more classes being filled, better accuracy can be achieved. It takes only a few minutes (about 2) to process all 1M images at 32 32 resolution using a single GPU. Notice that ImageNet models have been trained on 224 224 images, while here we test them on 32 32 noise images for the sake of computational complexity. A better approach would be to train the models on 32 32 images or feed the noise at 224 224 resolution. This, however, demands more computational power but may result in better performance.
|backbone||accuracy||run time||empty classes|
Overall, our pilot investigation on large scale datasets is promising. We believe better results than the ones reported in Table 1 are possible with further modifications (e.g., using better distance measures between an image and the average noise map for each class). Also, it is likely that increasing the number of samples will lead to better performance.
3.2 Adversarial Attack and Defense
Deep neural networks achieve remarkable results on various visual recognition tasks. They are, however, highly susceptible to being fooled by images that are modified in a particular way (so-called adversarial examples). Interesting adversarial examples are the ones that can confuse a model but not a human (i.e., imperceptible perturbations). Likewise, it is also possible to generate a pattern that is perceived by a human as noise but is classified by a network as a legitimate object with high confidence (Nguyen et al., 2015). Beyond the security implications, adversarial examples also provide insights into the weaknesses, strengths, and blind-spots of models.
Adversarial attack. A natural application of the bias maps is to utilize them to influence a black-box system, in targeted or un-targeted manners, by adding them to the healthy inputs. Over MNIST, we added different magnitudes of bias maps (controlled by ; Eq. 3) to the input digits and calculated the misclassification accuracy or fooling rate of a CNN (same as the one used in the previous section). This is illustrated in Fig. 6A. Obviously, there is a compromise between the perceptibility of perturbation (i.e., adding bias) and the fooling rate. With , we are able to manipulate the network to classify the augmented digit as the class of interest 21% of the time (Fig. 6C; chance is 10%). In comparison, adding the same amount of the mean image to digits fools the network almost always but is completely perceptible. In a similar vein, we are able to convert noise to a target digit class by adding bias to it (Fig. 6B). With , which is perceptually negligible (See supplement Fig. 17
), we can manipulate the network 20.7% of the time. Notice that in contrast to many black-box adversarial attacks that demand access to logits or gradients, our approach only requires the hard labels and does not make any assumption regarding the input distribution.
Adversarial defense. In a recent work, Brown et al. (2017) introduced a technique called adversarial patch as a backdoor attack on a neural network. They placed a particular type of pattern on some inputs and trained the network with the poisoned data. The patches were allowed to be visible but were limited to a small, localized region of the input image. Here, we explore whether and how classification images can be used to detect adversarial patch attacks.
We performed three experiments, two on MNIST and one on CIFAR-10 (Fig. 7). Over MNIST, we constructed two training sets as follows. In the first one, we took half of the 0s and placed a 33 patch (x-shape) on their top-left corner and relabeled them as 1. The other half of zeros and all other digits remained intact. In the second one, we placed a c-shape patch on the top-right corner of half of the 8s, relabeled them as 9, and left the other half and other digits intact. We then trained two 10-way CNNs (same architecture as in the previous section) on these training sets. The CNNs perform close to perfect on the healthy test sets. Over a test set with all zeros contaminated (or eights), they completely misclassify the perturbed digits (See confusion matrices in the 2nd and 4th rows of Fig. 7A). Computing the classification images for these classifiers, we find a strong activation at the location of the adversarial patches in both cases. Note that the derived classification images still resemble the ones we found using the un-attacked classifiers (Fig. 2A) but now new regions pop out. Over CIFAR-10, we placed an H-shape pattern on top-right of half of the birds and labeled them as cats. The trained CNN classifier performs normally on a clean dataset. Again, computing the bias unveils a tamper in the network (Fig. 7B). To verify these findings, we computed the average gradient of the classification loss with respect to the input image for intact and attacked networks over the healthy and tampered training sets (Fig. 7). The average gradient shows a slight activation at the location of the perturbation (Fig. 7D), but it is not as pronounced as results using bias images.
3.3 Filter visualization
A number of ways have been proposed to understand how neural networks work by visualizing their filters (Nguyen et al., 2019). Example approaches include plotting filters of the first layers, identifying stimuli that maximally activate a neuron, occlusion maps by masking image regions (Zeiler & Fergus, 2014), activation maximization by optimizing a random image to be classified as an object (Erhan et al., 2009), saliency maps by calculating the effect of every pixel on the output of the model (Simonyan et al., 2013), network inversion (Mahendran & Vedaldi, 2015), and network dissection (Bau et al., 2017). Here, we propose a new method based on spike triggered averaging.
For each model, we fed 1 million randomly generated patterns to the network and recorded the average response of single neurons at different layers. We changed the activation functions in the convolution layers of the CIFAR-10 CNN model totanh, as using ReLU activation resulted in some dead filters. Fig. 8 shows the results over MNIST and CIFAR-10 datasets. We also show the filters computed using real data for the sake of comparison in the supplement (Fig. 22). As it can be seen, filters extract structural information (e.g., oriented edges) and are similar to those often derived by other visualization techniques. Comparing derived filters using noise patterns and derived filtered using training on real data (i.e., kernel weights), we notice that the two are exactly the same. This holds over both MNIST and CIFAR-10 datasets (Fig. 22 in supplement).
Next, for the CIFAR-10 model, we computed mean layer activation maps of conv2, conv4, conv6, and fc layers by sending noise through the network. Results are shown in Fig. 20 in the supplement. Comparing these maps with the mean activation maps derived using real data, we observe high similarity in the fc layer and relatively less similarity in the other layers. The high similarity in the fc layer is because it is immediately before the class decision layer, and thus for a noise pattern to fall under a certain class, it has to have a similar weight vector as the learned weights from real data. This is corroborated by the higher average L2 distance across different classes in the fc layer, compared to the other layers, over both noise and real data (bottom panel in Fig. 20).
We then asked whether it is possible to bias the network towards certain classes (similar to the adversarial analysis in Fig. 6) by injecting information, learned from average noise patterns to the input image or its activation maps at different layers. For example, as shown in Fig. 7C, we can turn a frog into a car by adding the average conv6 activation of the noise patterns classified as a car to it. This can be done in a visually (almost) imperceptible manner. Results over other classes of CIFAR-10 and different activation layers are shown in Fig. 21 (supplement). For some classes (e.g., cat or bird), it is easy to impact the network, whereas for some others (e.g., horse) it is harder. Results indicate that for different objects, different layers have more influence on classification.
Microstimulation, the electrical current-driven excitation of neurons, is used in neurophysiology research to identify the functional significance of a population of neurons (Cohen & Newsome, 2004; Lewis et al., 2016). Due to its precise temporal and spatial characteristics, this technique is often used to investigate the causal relationship between neural activity and behavioral performance. It has also been employed to alleviate the impact of damaged sensory apparatus and build brain-machine interfaces (BMIs) to improve the quality of life of people who have lost the ability to use their limbs. For example, stimulation of the primary visual cortex creates flashes of light which can be used to restore some vision for blind people. Microstimulation has been widely used to study visual processing across several visual areas including MT, V1, V4, IT, and FEF (Moore & Fallah, 2004). Here, we investigate how augmenting the stimuli with white noise impacts internal activations of artificial neural networks and their outputs.
We linearly combined signal and white noise, according to Eq. 3, and measured the classification accuracy of a CNN trained on MNIST (Fig. 9). Without any stimulation, with the original network biases and weights, increasing the amount of signal (shown on the x-axis) improves the accuracy from 0 (corresponding to 100% noise) to 1 (corresponding to 100% signal). The resulting S-shaped curve resembles the psychometric functions observed in human psychophysics experiments (Wichmann & Hill, 2001). We then varied the amount of network bias in different layers according to the following formula and measured the accuracy again:
where is the bias term for map in layer , and is the activation of neuron at the th map of the th layer. controls the magnitude of stimulation. is used to scale the activation values, since sensitivity of the output to neurons at different layers varies (here we use for fc, conv1, and conv2, respectively). Bias term () is shared across all neurons in a map (i.e., for the same kernel). Notice that increasing bias in Eq. 6 is proportional to the map activation. Thus, stimulation has a higher impact on more active (selective) neurons.
Increasing bias of the fc neurons shifts the psychometric function to the left. This means that for the same amount of noise as before (i.e., no stimulation), now CNN classifies the input more frequently as the target digit. In other words, the network thinks of noise as the digit. Increasing fc biases consistently elevates accuracy for all digits. Conversely, reducing the fc bias shifts the psychometric function to the right for all digits (Fig. 24 in supplement; i.e., using minus sign in Eq. 6). The effect of stimulation on convolutional layers is not consistent. For example, increasing conv2 bias shifts the curves to the right for 0 and 1, and to the left for 3. We observed that stimulation or inhibition of conv1 layer almost always hurts all digits. We speculate this might be because conv1 filters capture features that are shared across all digits, and thus a subtle perturbation hurts the network.
We were able to replicate the above results using a binary CNN akin to yes/no experiments on humans or monkeys. Results are provided in supplement Fig. 25. Our findings qualitatively agree with the results reported in Afraz et al. (2006). They artificially stimulated clusters of IT neurons while monkeys judged whether noisy visual images were ‘face’ or ‘non-face’. Microstimulation of face-selective neurons biased the monkeys’ decisions towards the face category.
4 Discussion and Conclusion
We showed that white noise analysis is effective in unveiling hidden biases in deep neural networks and other types of classifiers. A drawback is a need for a large number of trials. To lower the sample complexity, we followed the approach in Greene et al. (2014) and also recruited generative models. As a result, we were able to lower the sample complexity dramatically. As another alternative, Vondrick et al. (2015) used the Hoggles feature inversion technique (Vondrick et al., 2013) to generate images containing subtle scene structures. Their computed bias maps roughly resembled natural scenes. We found that the quality of the bias maps highly depends on the classifier type and the number of trials. Also, classification images over natural scene datasets are not expected to look like the instances of natural images since even the mean images do not represent sharp objects. Please see Figs. 2 and 5. In this regard, spike triggered covariance can be utilized to find stimuli (eigen vectors) to which a network or a neuron responds (Schwartz et al., 2006).
We foresee several avenues for future research. We invite researchers to employ the tools developed here to analyze even more complex CNN architectures including ResNet (He et al., 2016) and InceptionNet (Szegedy et al., 2017). They can also be employed to investigate biases of other models such as CapsuleNets (Hinton et al., 2018) and GANs (Goodfellow et al., 2014a), and to detect and defend against other types of adversarial attacks. The outcomes can provide a better understanding of the top-down processes in deep networks, and the ways they can be integrated with bottom-up processes. Moreover, applying some other methods from experimental neuroscience (Bickle, 2016) (e.g., lesioning, staining) and theoretical neuroscience (e.g., spike-triggered non-negative matrix factorization (Liu et al., 2017), Bayesian STC (Park & Pillow, 2011), and Convolutional STC (Wu et al., 2015)) to inspect neural networks is another interesting future direction. Using classification images to improve the accuracy of classifiers (as in Vondrick et al. (2015)) or their robustness (as was done here) are also promising directions.
Here, we focused primarily on visual recognition. Rajashekar et al. (2006) used classification images to estimate the template that guides saccades during the search for simple visual targets, such as triangles or circles. Caspi et al. (2004) measured temporal classification images to study how the saccadic targeting system integrates information over time. Keane et al. (2007) utilized classification images to investigate the perception of illusory and occluded contours. Inspired by these works, classification images, and STA can be applied to other computer vision tasks such as object detection, edge detection, activity recognition, and segmentation. Finally, unveiling biases of complicated deep networks can be fruitful in building bias-resilient and fair ANNs (e.g., racial fairness).
In summary, we utilized two popular methods in computational neuroscience, classification images and spike triggered averaging, to understand and interpret the behavior of artificial neural networks. We demonstrated that they bear value for practical purposes (e.g., solving challenging issues such as adversarial attacks) and for further theoretical advancements. More importantly, our efforts show that confluence across machine learning, computer vision, and neuroscience can benefit all of these fields (See Hassabis et al. (2017)). We will release our code and data to facilitate future research.
- Afraz et al. (2006) Seyed-Reza Afraz, Roozbeh Kiani, and Hossein Esteky. Microstimulation of inferotemporal cortex influences face categorization. Nature, 442(7103):692, 2006.
- Ahumada Jr (1996) AJ Ahumada Jr. Perceptual classification images from vernier acuity masked by noise. Perception, 25(1_suppl):2–2, 1996.
- Bashivan et al. (2019) Pouya Bashivan, Kohitij Kar, and James J DiCarlo. Neural population control via deep image synthesis. Science, 364(6439):eaav9436, 2019.
Bau et al. (2017)
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba.
Network dissection: Quantifying interpretability of deep visual
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6541–6549, 2017.
- Bickle (2016) John Bickle. Revolutions in neuroscience: Tool development. Frontiers in systems neuroscience, 10:24, 2016.
Borji & Itti (2012)
Ali Borji and Laurent Itti.
State-of-the-art in visual attention modeling.IEEE transactions on pattern analysis and machine intelligence, 35(1):185–207, 2012.
- Brown et al. (2017) Tom B Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer. Adversarial patch. arXiv preprint arXiv:1712.09665, 2017.
- Caspi et al. (2004) Avi Caspi, Brent R Beutter, and Miguel P Eckstein. The time course of visual information accrual guiding eye movement decisions. Proceedings of the National Academy of Sciences, 101(35):13086–13090, 2004.
- Cohen & Newsome (2004) Marlene R Cohen and William T Newsome. What electrical microstimulation has revealed about the neural basis of cognition. Current opinion in neurobiology, 14(2):169–177, 2004.
- Dayan et al. (2001) Peter Dayan, Laurence F Abbott, et al. Theoretical neuroscience, volume 806. Cambridge, MA: MIT Press, 2001.
- DeAngelis et al. (1993) Gregory C DeAngelis, Izumi Ohzawa, and RD Freeman. Spatiotemporal organization of simple-cell receptive fields in the cat’s striate cortex. ii. linearity of temporal and spatial summation. Journal of Neurophysiology, 69(4):1118–1135, 1993.
- Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Ieee, 2009.
- Erhan et al. (2009) Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. University of Montreal, 1341(3):1, 2009.
- Fong et al. (2018) Ruth C Fong, Walter J Scheirer, and David D Cox. Using human brain activity to guide machine learning. Scientific reports, 8(1):5397, 2018.
- Goodfellow et al. (2014a) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pp. 2672–2680, 2014a.
- Goodfellow et al. (2014b) Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014b.
- Greene et al. (2014) Michelle R Greene, Abraham P Botros, Diane M Beck, and Li Fei-Fei. Visual noise from natural scene statistics reveals human scene category representations. arXiv preprint arXiv:1411.5331, 2014.
Hassabis et al. (2017)
Demis Hassabis, Dharshan Kumaran, Christopher Summerfield, and Matthew
Neuroscience-inspired artificial intelligence.Neuron, 95(2):245–258, 2017.
- He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
- Hinton et al. (2018) Geoffrey E Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with em routing. In 6th international conference on learning representations, ICLR, 2018.
- Jones & Palmer (1987) Judson P Jones and Larry A Palmer. The two-dimensional spatial structure of simple receptive fields in cat striate cortex. Journal of neurophysiology, 58(6):1187–1211, 1987.
Keane et al. (2007)
Brian P Keane, Hongjing Lu, and Philip J Kellman.
Classification images reveal spatiotemporal contour interpolation.Vision Research, 47(28):3460–3475, 2007.
- Kingma & Welling (2013) Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
- Krizhevsky et al. (2009) Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Krizhevsky et al. (2012)
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.
Imagenet classification with deep convolutional neural networks.In Advances in neural information processing systems, pp. 1097–1105, 2012.
- LeCun et al. (1998) Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
- Lewis et al. (2016) Philip M Lewis, Richard H Thomson, Jeffrey V Rosenfeld, and Paul B Fitzgerald. Brain neuromodulation techniques: a review. The neuroscientist, 22(4):406–421, 2016.
- Liu et al. (2017) Jian K Liu, Helene M Schreyer, Arno Onken, Fernando Rozenblit, Mohammad H Khani, Vidhyasankar Krishnamoorthy, Stefano Panzeri, and Tim Gollisch. Inference of neuronal functional circuitry with spike-triggered non-negative matrix factorization. Nature communications, 8(1):149, 2017.
- Mahendran & Vedaldi (2015) Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5188–5196, 2015.
- Marmarelis (2012) Vasilis Marmarelis. Analysis of physiological systems: The white-noise approach. Springer Science & Business Media, 2012.
- Meister et al. (1994) Markus Meister, Jerome Pine, and Denis A Baylor. Multi-neuronal signals from the retina: acquisition and analysis. Journal of neuroscience methods, 51(1):95–106, 1994.
- Moore & Fallah (2004) Tirin Moore and Mazyar Fallah. Microstimulation of the frontal eye field and its effects on covert spatial attention. Journal of neurophysiology, 91(1):152–162, 2004.
- Murray (2011) Richard F Murray. Classification images: A review. Journal of vision, 11(5):2–2, 2011.
- Nguyen et al. (2015) Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 427–436, 2015.
- Nguyen et al. (2019) Anh Nguyen, Jason Yosinski, and Jeff Clune. Understanding neural networks via feature visualization: A survey. arXiv preprint arXiv:1904.08939, 2019.
- Park & Pillow (2011) Il Memming Park and Jonathan W Pillow. Bayesian spike-triggered covariance analysis. In Advances in neural information processing systems, pp. 1692–1700, 2011.
- Rajashekar et al. (2006) Umesh Rajashekar, Alan C Bovik, and Lawrence K Cormack. Visual search in noise: Revealing the influence of structural cues by gaze-contingent classification image analysis. Journal of Vision, 6(4):7–7, 2006.
- Recht et al. (2019) Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? arXiv preprint arXiv:1902.10811, 2019.
- Reid & Alonso (1995) R Clay Reid and Jose-Manuel Alonso. Specificity of monosynaptic connections from thalamus to visual cortex. Nature, 378(6554):281, 1995.
- Sakai & Naka (1987) HIROKO M Sakai and K Naka. Signal transmission in the catfish retina. v. sensitivity and circuit. Journal of Neurophysiology, 58(6):1329–1350, 1987.
- Sandler & Marmarelis (2015) Roman A Sandler and Vasilis Z Marmarelis. Understanding spike-triggered covariance using wiener theory for receptive field identification. Journal of vision, 15(9):16–16, 2015.
- Schwartz et al. (2006) Odelia Schwartz, Jonathan W Pillow, Nicole C Rust, and Eero P Simoncelli. Spike-triggered neural characterization. Journal of vision, 6(4):13–13, 2006.
- Simonyan et al. (2013) Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
- Szegedy et al. (2013) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
Szegedy et al. (2017)
Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi.
Inception-v4, inception-resnet and the impact of residual connections on learning.In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
- Vondrick et al. (2013) Carl Vondrick, Aditya Khosla, Tomasz Malisiewicz, and Antonio Torralba. Hoggles: Visualizing object detection features. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1–8, 2013.
- Vondrick et al. (2015) Carl Vondrick, Hamed Pirsiavash, Aude Oliva, and Antonio Torralba. Learning visual biases from human imagination. In Advances in neural information processing systems, pp. 289–297, 2015.
- Wichmann & Hill (2001) Felix A Wichmann and N Jeremy Hill. The psychometric function: I. fitting, sampling, and goodness of fit. Perception & psychophysics, 63(8):1293–1313, 2001.
- Wu et al. (2015) Anqi Wu, Il Memming Park, and Jonathan W Pillow. Convolutional spike-triggered covariance analysis for neural subunit models. In Advances in neural information processing systems, pp. 793–801, 2015.
- Xiao et al. (2017) Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
- Zeiler & Fergus (2014) Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818–833. Springer, 2014.
Appendix A Appendix
Creating visual noise from natural scene statistics. We followed Greene et al. (2014) to generate noise patterns containing subtle structures. We amassed a digit database of 60K images from MNIST training set and represented each image as the output of a bank of Gabor filters at three spatial scales (2, 4 and 10 cycles per image, and wavelets were truncated to lie within the borders of the image), four orientations (0, 45, 90 and 135 degrees) and two quadrature phases (0 and 90 degrees). Thus, each image is represented by total Gabor wavelets. Weights of Gabor wavelets for each image were determined using ridge regression. We then performed principal components analysis (PCA) on the 60K-image by the 960-wavelet weight matrix. We kept the first 250 principal components that explain 96.1% of the variance in data. A noise image was created by choosing a random value for each principal component score, scaled to the observed range for each component.
We also gathered a natural object database of 50K images from the CIFAR-10 training set. For these colored images, we performed the above-mentioned approach on each channel. More specifically, we represented each 32 32-sized channel with four-scale (2, 4, 7, 11 cycles), four-orientation, two-phase Gabor wavelets, which results in total Gabor wavelets per channel. Then for each channel, we performed ridge regression to get the 50K-by-1520 weight matrix, which is passed to PCA and kept the first 600 PCs. These PCAs can explain variance in three channels as 97.57%, 97.51%, and 97.52%, respectively.