have shown promise for classification tasks in computer vision. However, learning a model using small data,e.g. annotations on 1–3 examples, is still a great challenge for state-of-the-art algorithms. Without sufficient annotations, the conventional end-to-end learning usually has no mechanism to ensure that the CNN actually learns correct knowledge, rather than over-fit to noisy signals.
Therefore, instead of letting the CNN “guess” new knowledge from small training data, we aim to mine certain patterns from a well pre-trained CNN to represent semantic parts of the object for part localization. We notice that when a CNN is well pre-trained/finetuned using a large number of object-box annotations for object classification, the CNN has encoded massive implicit patterns for local object shapes in its conv-layers. Each pattern may represent a local shape of an object.
In this paper, we propose to incorporate human interactions into the mining of part patterns, in order to ensure we use correct patterns to represent the target semantic part. As shown in Fig. 1, we are given a large number of images with object-box annotations to train a CNN for category classification, but very few (e.g. 1–3) objects of a category have annotations of a semantic part. We mine hundreds of latent patterns from the pre-trained CNN as candidates that potentially represent the target part. Then, we visualize the mined patterns and ask human users to manually select semantically correct latent patterns to build up the model for part localization, just like playing LEGO blocks.
More specifically, during the process of pattern mining, each latent pattern is expected 1) to frequently appear on objects of the category, 2) to be strongly activated by a compositional (or contextual) shape w.r.t the target part, and 3) to keep good spatial relationship with other latent patterns. Because all latent patterns are well pre-trained using massive data, these patterns represent common shapes of a category, instead of being over-fitted to a few part annotations.
For human interactions, we require human users to remove patterns corresponding to background noises and specificity of certain samples. Because we allow people to directly point out model flaws, this interactive-learning strategy is more effective than end-to-end “guessing” true knowledge of the object part from small training data.
And-Or graph representation:
Before conducting human interactions, we need to represent latent patterns at the semantic level for pattern manipulation, rather than at the level of CNN neural units. Note that the same object part (e.g.
the head) in different images may appear in different image positions, thereby activating neurons in different feature map positions. Thus, given a feature map of a filter in a conv-layer, we need to first remove noisy activations from the feature map, and then summarize the rest neural activations on numerous neural units into a few latent patterns for human interactions. When we infer a latent pattern in different images, the pattern may appear in different feature map positions due to object deformation. Moreover, a filter’s feature map may be activated by multiple object parts (e.g. being activated by both the head and the leg), we need to disentangle latent patterns of different object parts from the same feature map.
Therefore, we use an And-Or graph (AOG) to clearly represent the semantic hierarchy of the patterns that are mined from conv-layers. As shown in Fig. 2, we build a four-layer AOG to represent the semantic hierarchy ranging from semantic part, part templates, latent patterns, to CNN units. We use AND nodes in the AOG to encode compositional regions of a part, and use OR nodes to encode a list of alternative appearances/deformations for a local region.
Based on the AOG, we localize patterns on CNN feature maps and visualize these patterns. Then, users can remove irrelevant patterns by pruning certain AOG nodes.
Pattern visualization and interactions:
We apply the up-convolutional neural network (up-conv-net) in  to visualize latent patterns in the AOG. We train different up-conv-nets to visualize latent patterns corresponding to different conv-layers of the CNN. According to visualization results, latent patterns in low conv-layers usually describe object details, and those in high conv-layers mainly correspond to large-scale parts or contexts. Therefore, we use low-layer patterns to represent details within the target part, and require users to remove low-layer patterns outside the part. We select patterns in high conv-layers to represent the contextual information of the target part.
Method generality: Our method is a general solution to interactively learning object parts based on pre-trained CNNs. First, the AOG representation of objects  can be compatible with different features. Second, our method has been tested on AOGs with two different types of neural patterns [19, 18].
Instead of end-to-end learning new information from training data, this study explores the probability of directly selecting certain patterns from a pre-trained CNN to build a model for part localization. Our method mines middle-level latent patterns from the CNN and incorporates human interactions to manually select correct patterns. Our method exhibited superior performance in experiments, which demonstrates the effectiveness of human interactions in weakly-supervised learning.
2 Related work
CNN visualization, semanticization, and interactions:
In recent years, many methods have been developed to explain the semantics hidden in the CNN. Studies of [17, 10, 14] passively visualized content of some given CNN units.  analyzed statistics of CNN features.
Unlike passive CNN visualization, we hope to actively semanticize CNNs by discovering patterns related to the target part, which is more challenging. Given CNN feature maps, Zhou et al. [20, 21] discovered latent “scene” semantics. Simon et al. discovered objects  from CNN activations in an unsupervised manner, and learned part concepts in a supervised fashion .  mined CNN patterns for a part concept and transformed the pattern knowledge into an AOG model.
However, without sufficient supervision, previous studies cannot ensure the extracted CNN patterns to have correct part semantics. Thus, we visualize candidate patterns and require users to manually select correct ones, so as to create a better “white-box” explanation of the target part.
In many studies, people used AOGs to represent the hierarchical semantic hierarchy of objects or scenes [22, 11]. We use the AOG to associate the latent patterns with part semantics, which eases the visualization of CNN patterns and enables semantic-level interactions on CNN patterns.
Unsupervised learning of objects and parts:
Unsupervised object discovery  was formulated as a problem of mining common foreground patterns from images, and many sophisticated methods were developed for this problem. Whereas, given a pre-trained object-level model, un-/weakly-supervised learning of part representations is a different problem.
In this section, we propose a general method to interactively learn object-part models. In fact, there are a number of techniques to mine neural patterns from CNNs to represent middle-level object shapes. Each of these patterns can be organized using a widely used AOG model. A clear AOG representation allows human users to refine the model by interactively modifying the AOG structure. The basic idea is to visualize latent patterns in the AOG and to let human users identify and remove latent patterns that are not closely related to the target part.
3.1 Preliminaries: AOG representation
The AOG has been a typical object representation for years [11, 22]. Here, we briefly introduce the AOG structure, which has been widely used to represent neural patterns of CNNs in studies of  and .
The AOG organizes latent patterns hidden in the CNN to explain the semantic hierarchy of an object part. We use the AOG to parse object parts from images. As shown in Fig. 2, we use a four-layer AOG hierarchy ranging from semantic part (OR node), part templates (AND nodes), latent patterns (OR nodes), to CNN units (terminals). In the AOG, an OR node encodes a list of alternative candidates as children, while an AND node uses each of its children to describe a certain compositional shape (or a contextual area) of the father node.
Given an object image 222Considering the CNN’s superior performance in object detection, as in , object detection and part localization are considered two separate processes for evaluation. Thus, we crop to only contain the object and resize for CNN inputs to simplify the scenario of learning for part localization., we use the AOG for part parsing. I.e. we first use the CNN to compute ’s feature maps of its conv-layers, and then determine a parse tree within the AOG to explain neural activations on the feature maps and simultaneously localize the target part.
As red lines in Fig. 2, during the parsing procedure, we 1) select a certain part template (AND node in the 2nd Layer) to explain the target part (root OR node in the AOG), 2) parse an image region for the part template (i.e. part localization), and 3) for each latent pattern (OR node in the 3rd Layer) under the part template, determine a CNN unit (terminal node) within a deformation range to localize the local shape of this latent pattern. To be precise, we achieve the part parsing in a bottom-up manner. I.e. in the beginning, we compute an inference score for each terminal node (CNN unit), and then propagate these scores up to nodes of latent patterns and part templates following certain And-Or rules for part parsing.
The top node of “semantic part” (OR node) encodes a number of alternative part templates as children, each denoted by . Each part template naturally corresponds to a type of part appearance observed from a certain perspective. Given parsing results of all part templates, the top node selects the part template with the highest inference score as the true parsing configuration.
where denotes the overall inference score on image , and represents the image region parsed for part template . measures the inference score when we parse an image region for the part template .
Then, each part template (AND node) uses children latent patterns to represent its local compositional shapes or contextual area. Thus, we can formulate ’s inference score as the sum of its children’s inference scores, i.e. , where is the children set of , and denotes the inference score of .
Finally, each latent pattern (OR node) naturally corresponds to a square deformation range within a certain conv-slice/channel of the CNN. All CNN units within 333We set a constant deformation range for each latent pattern, which covers -by- of the feature map of the conv-slice. Deformation ranges of different patterns in the same conv-slice may overlap. The central position of , , is a parameter to estimate.
, is a parameter to estimate.are regarded as deformation candidates of . Just following the OR-node logic in Eq. (1), also selects the best child unit as the true parsing configuration, , . The selected child propagates both its score and parsed region to parent . The image region for each CNN unit is fixed, i.e. we simply propagate the receptive field of to the image plane and obtain . Therefore, we can re-write the above And-Or logics as a DPM-like (deformable part model) model:
where the part template score comprises latent pattern scores, and each latent pattern selects its best CNN unit as the parsing configuration (i.e. computing the image region corresponding the CNN unit). The unary term 444 measures both the neural response of the CNN unit corresponding to the position of and the score for local deformation w.r.t. ’s ideal position , where returns the center position. , where denotes the displacement from to . is computed based on part annotations. , . measures the local inference quality, and the pairwise term 44footnotemark: 4 encodes spatial relationship between part template and latent pattern .
3.2 Latent patterns
In order to demonstrate the generality of the proposed method, we use two different types of latent patterns mined from CNNs to construct different AOGs. Our method allows users interactively select different latent patterns for learning. The first type of patterns are proposed in  as local middle-level features of objects, namely middle-level patterns. The method of  is proposed to learn an explanatory graph to explain the knowledge hierarchy inside a pre-trained CNN. We use nodes in the explanatory graph as the second type of patterns, namely explanatory patterns. In experiments, we compared part-localization performance of AOGs based on different patterns.
The experimental settings for the mining of the above two patterns are the same, which can be summarized as follows. The input is a set of cropped object images of a category, denoted by , where only a few objects have ground-truth annotations of the target part. Each part annotation on image includes both the ground-truth part template and the true bounding box of the part
. The CNN is pre-trained to classify object images inof a category from random images. Given the part annotations, [19, 18] mine middle-level patterns and explanatory patterns from conv-layers in the CNN to represent neural activations that are highly related to the part annotations. We use the two patterns to build two types of AOGs.
3.3 AOG visualization
We learn up-conv-nets  to roughly555Up-conv-nets  cannot ensure a “strict” correspondence between each CNN unit and its visualized appearance. visualize the content within AOG nodes. Given a feature map of a CNN’s conv-layer as input, the up-conv-net was originally proposed to invert the CNN feature map and output the image corresponding to the feature map, namely, image reconstruction. The AOG encodes several part templates of a semantic part, and each part template consists of latent patterns in different conv-layers. Thus, in each step, we visualize and refine latent patterns of a certain part template. We extend the up-conv-net  to visualize latent patterns within the sub-AOG under a given part template .
Given a training object sample whose part is explained by the part template in the AOG, we use ’s sub-graph to localize the target part based on Eq. (2). During the part-localization process, each latent pattern is assigned with a certain CNN unit. Thus, we can use activation responses of the corresponding CNN units to reconstruct the patterns’ appearance. We implement the image reconstruction by simply filtering out all unrelated activation responses from the CNN feature map (i.e. setting activations to zero) and using the up-conv-net to invert the modified feature map.
Note that latent patterns of each part template are extracted from different conv-layers of the CNN, and they potentially represent local compositional shapes at different scales. As shown in Fig. 3, latent patterns in lower conv-layers usually represent small-scale details (e.g. edges and corners), but these patterns are usually not discriminative enough and more likely to be activated by background noises. In contrast, latent patterns in higher conv-layers mainly correspond to large-scale appearances/contexts without encoding much object details, but they usually consistently represent certain parts among different objects.
As a result, we train different up-conv-nets using feature maps of different conv-layers for image reconstruction, in order to avoid detailed object appearance being mixed with large-scale object patterns during image reconstruction. In this way, we apply different human-interaction strategies for latent patterns in different conv-layers (which will be introduced later).
Training and using up-conv-nets:
Given a CNN that is finetuned for a category, we train an up-conv-net for each conv-layer of the CNN. Training samples are the cropped object images of the category. Given the object images, we use the CNN to extract feature maps of the target conv-layer as the input of the up-conv-net. Output of the up-conv-net is pixel-level image-reconstruction results. The up-conv-net is trained based on the loss of the L-2 distance between the original image and the reconstructed image :
When we apply the up-conv-net to AOG visualization, we use the AOG to localize the target part on the given object . Let the part-parsing process localize a latent pattern at the horizontal coordinate in -th conv-slice of the target conv-layer. We consider that neural responses at in this conv-slice are related to 666For conv-layers of VGG-16 whose feature map size , we select neural responses within .. We set all neural responses unrelated to any latent patterns to zero in the feature map. Then, we use the up-conv-net to invert the modified feature map for image reconstruction.
3.4 Human interactions
Based on visualization results of latent patterns, we further use human interactions to remove latent patterns that are not closely related to the target part. The flowchart of human interactions is designed as follows. Given an part template , our method sequentially produces different image-reconstruction results based on ’s latent patterns extracted from different conv-layers. Then, given the reconstructed image w.r.t. each conv-layer, we require people to annotate image regions that do not contribute to the localization of the target part. Let denote all image area of , and denote the union of the annotated image regions. A latent pattern under is considered not related to the target part and removed from the AOG, if 1) ’s parsed image region localizes within , and 2) it satisfies
where denotes the value of a pixel in . is equal to the gradient of pattern score w.r.t. pixel (see  for details of the gradient computation). Pixels with high gradients usually have high correspondence to pattern . Compared to the above equation, the up-conv-net can only show rough image regions of latent patterns.
Rules for human interactions
We believe that the localization of a certain semantic part mainly relies on two types of information: 1) the contextual information (e.g. the global pose of the entire object) for rough part localization from a global view, and 2) detailed part appearance for accurate localization. In general, patterns in low conv-layers describe object details, and those in high conv-layers represent contextual information. Therefore, as shown in Fig. 3, for latent patterns in high conv-layers, we mainly remove those localized on the background outside the “object.” For latent patterns in low conv-layers, we usually remove those outside “part bounding boxes.”
4.1 Implementation details
We learned the AOG based on a 16-layer VGG network (VGG-16) 
, which was pre-trained as follows. The VGG-16 was first pre-trained using the 1.3M images in the ImageNet ILSVRC 2012 dataset with a loss for 1000-category classification. Then, we further finetuned the VGG-16 using cropped object images in a category to classify target objects from background images. Just as in , we selected the last nine conv-layers in the VGG-16 as valid layers, and extracted latent patterns from these conv-layers to build the AOG.
Note that feature maps for Conv-layers 5–7, Conv-layers 8–10, and Conv-layers 11–13 are three , three , and three matrices, respectively. To simplify the system, we merged conv-slices in Conv-layers 5–7, which contained latent patterns, into a new feature map of ( denotes the number of valid conv-slices in Conv-layers 5–7). We learned a up-conv-net, namely Net-1, to visualize latent patterns in the new feature map. Similarly, we learned Net-2 and Net-3 for Conv-layers 8–10 and Conv-layers 11–13, respectively.
In order to make a comprehensive evaluation of the proposed method, we chose three benchmark datasets for testing, i.e. the Pascal VOC Part dataset , the CUB200-2011 dataset , and the ILSVRC 2013 DET Animal-Part dataset . Note that as in many previous studies [3, 19], we chose animal categories in these three datasets to evaluate part-localization performance, because animals usually contain multiple non-rigid parts, which presents a key challenge for part localization.
|Fast-RCNN (1 ft) ||N||0.313||0.370||0.250||0.318||0.375||0.343||0.291||0.365||0.287||0.321||0.291||0.305||0.349||0.261||0.290||0.165|
|Fast-RCNN (2 fts) ||Y||0.355||0.382||0.421||0.298||0.413||0.168||0.328||0.383||0.298||0.219||0.196||0.217||0.245||0.265||0.264||0.220|
|Fast-RCNN (1 ft) ||N||0.372||0.360||0.302||0.289||0.342||0.266||0.220||0.349||0.328||0.334||0.351||0.261||0.337||0.328||0.311|
|Fast-RCNN (2 fts) ||Y||0.355||0.324||0.275||0.266||0.292||0.235||0.212||0.271||0.329||0.343||0.259||0.163||0.285||0.246||0.284|
In experiments, we used our interactive learning method to refine the AOG learned by , namely the Ours method. Then, we followed the algorithm of  to further build a new AOG, which used explanatory patterns in  as latent patterns. We conducted the interactive learning on the new AOG, and named it the Ours (explanatory patterns) method.
We compared our methods with a total of fourteen methods in part localization. The baselines included 1) state-of-the-art algorithms for object detection (i.e. directly detecting target parts from objects), 2) conventional graphical/part models for part localization (modeling both the part appearance and the relationships with parts and objects), and 3) the methods selecting CNN patterns to describe object parts.
The first baseline was the standard method of using AOGs for part localization without applying human interactions , namely Mining-raw. The comparison with Mining-raw demonstrated the effectiveness of human interactions in learning.
Then, two baselines were implemented based on the Fast-RCNN . We finetuned the fast-RCNN with a loss for detecting a single class/part from background, rather than for multi-class/part detection to ensure a fair comparison. The first baseline was the standard fast-RCNN, namely Fast-RCNN (1 ft), which directly finetuned the VGG-16 network to detect parts on objects. We annotated object parts on well cropped object images as training samples (objects had been cropped using their bounding boxes). Then, for the second baseline, namely Fast-RCNN (2 fts), we slightly modified original algorithm of Fast-RCNN. Given the large number of object-box annotations in the target category, the Fast-RCNN (2 fts) first finetuned the VGG-16 network with the loss of “detecting objects from entire images.” Then, given a small number of part annotations, Fast-RCNN (2 fts) further finetuned the VGG-16 to detect parts from objects. This two-step finetuning made a full use of the massive object-level annotations, which enabled a fair comparison.
|Fast-RCNN (1 ft) ||N||0.4517|
|Fast-RCNN (2 fts) ||Y||0.4131|
|Ours (explanatory patterns)||Y||0.0860|
|Fast-RCNN (1 ft)A||0.324||0.324||0.325||0.272||0.347||0.314||0.318|
|Fast-RCNN (2 fts)A||0.350||0.295||0.255||0.293||0.367||0.260||0.303|
|Ours (explanatory patterns)A||0.162||0.128||0.258||0.137||0.179||0.187||0.175|
Another typical baseline was CNN-PDD proposed in . CNN-PDD selected certain conv-slices (channels) of a CNN to represent the target part for part localization. In CNN-PDD, the CNN was pre-trained using 1.3M images in the ImageNet ILSVRC 2012 dataset. Just like Fast-RCNN (1 ft), we also extended the method of  as a new baseline CNN-PDD-ft, in order to make a full use of object-level annotations. CNN-PDD-ft pre-finetuned the VGG-16 network using object-box annotations of the target category before the selection of CNN channels.
We also compared our method with two DPM-related methods, i.e. the strongly supervised DPM (SS-DPM-Part)  and the technique in  (PL-DPM-Part), respectively. These two methods trained DPMs with part annotations for part localization. The next baseline, namely Part-Graph, used a graphical model for part localization .
In the scope of weakly-supervised learning, “simple” methods are usually insensitive to the over-fitting problem. Therefore, we selected six baselines, which extracted fc7 features for each image patch and used a SVM for part detection. The first three baselines were original proposed in , namely fc7+linearSVM, fc7+RBF-SVM, and fc7+NN, which used a linear SVM, a RBF-SVM, and the nearest-neighbor strategy, respectively, for part detection. The other three baselines combined both the fc7 feature and the spatial position () of each image patch as the final feature. We used fc7+sp+linearSVM, fc7+sp+RBF-SVM, and fc7+sp+NN to denote the last three baselines.
All the baselines were provided with object bounding boxes and used the same set of part annotations for training to ensure a fair comparison.
|bird beak||bird neck||bird wing||cat eye|
|cow ear||dog nose||dog eye||horse ear|
|horse eye||sheep neck||sheep muzzle||sheep eye|
4.4 Evaluation metric
 mentioned that it is necessary to remove factors of object detection from the evaluation of part localization. Therefore, as in , original images were cropped using object bounding boxes for testing to make a fair comparison. All the baselines used the cropped images for part localization/detection. In addition, some baselines for part detection may predict more than one locations for a part. Like in [13, 3], we took the detected part with the highest confidence in each image as the localization result. We evaluated part-localization performance using the metric of normalized distance, which has been widely used [13, 19]. Given an object, the normalized distance is the Euclidean distance between predicted part center and ground-truth part center, divided by the diagonal length of the object bounding box.
4.5 Experimental results and analysis
In experiments, we learned AOGs for the head, neck, nose, muzzle, beak, eye, ear, and wing parts of the six animal categories in the Pascal VOC Part dataset. For the ILSVRC 2013 DET Animal-Part dataset and the CUB200-2011 dataset, we learned an AOG for the head part777It is the “forehead” part for birds in the CUB200-2011 dataset. of each category. Because the head is shared by all categories in the two datasets, we selected the head as the target part to enable a fair comparison.
For each of the 37 object categories in the above three benchmark datasets, the experimental setting was the same as in . Each part concept of the category was defined to have three different part templates, and a single part box was annotated for each part template. I.e. we used a total of three annotations to build the AOG for the part. All the baselines learned models using the same three part annotations. On average, labeling a bounding box cost 3.4 seconds (i.e. 6.8 seconds for labeling both the object and the part), and the average time of human interactions per image was 12.3 seconds.
Fig. 4 shows patterns before and after human interactions. Fig. 5 shows part-localization results based on AOGs. Tables 1, 2 and 3 compare part-localization performance of different methods on the ILSVRC 2013 DET Animal-Part dataset, the CUB200-2011 dataset, and the Pascal VOC Part dataset, respectively. Table 4 lists localization results of various object parts in the Pascal VOC Part dataset before and after human-interactions. Our method exhibited superior performance to other baselines.
5 Conclusions and discussion
In this paper, we attempted to use human interactions to manually correct the representation of a semantic part, in the scenario of weak-supervised learning. We proposed to build a model for a semantic part by selecting related latent patterns from pre-trained CNNs and removing irrelevant patterns based on human subjective perception. We successfully mined and transferred latent patterns from a pre-trained CNN to a human-interpretable AOG model, which eased the visualization of latent patterns and allowed people to directly manipulate these patterns in the AOG. We can parallel this learning process to building LEGO blocks.
With the help of human interactions, our method ensured that the model used semantically correct patterns to represent an object part. This is different from conventional batch learning methods that usually let the computer “guess” the target knowledge representation from training samples. Our method exhibited superior performance in experiments, which demonstrated the effectiveness of incorporating human interactions in weakly-supervised learning.
M. Aubry and B. C. Russell.
Understanding deep features with computer-generated imagery.In ICCV, 2015.
-  H. Azizpour and I. Laptev. Object detection using strongly-supervised deformable part models. In ECCV, 2012.
-  X. Chen, R. Mottaghi, X. Liu, S. Fidler, R. Urtasun, and A. Yuille. Detect what you can: Detecting and representing objects using holistic models and body parts. In CVPR, 2014.
-  J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
-  A. Dosovitskiy and T. Brox. Inverting visual representations with convolutional networks. In CVPR, 2016.
-  R. Girshick. Fast r-cnn. In ICCV, 2015.
-  A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
-  Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, 1998.
-  B. Li, W. Hu, T. Wu, and S.-C. Zhu. Modeling occlusion by discriminative and-or structures. In ICCV, 2013.
-  A. Mahendran and A. Vedaldi. Understanding deep image representations by inverting them. In CVPR, 2015.
-  Z. Si and S.-C. Zhu. Learning and-or templates for object recognition and detection. In PAMI, 2013.
-  M. Simon and E. Rodner. Neural activation constellations: Unsupervised part model discovery with convolutional networks. In ICCV, 2015.
-  M. Simon, E. Rodner, and J. Denzler. Part detector discovery in deep convolutional neural networks. In ACCV, 2014.
-  K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In arXiv:1312.6034v2, 2013.
-  K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
-  C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-2011-001, In California Institute of Technology, 2011.
-  M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014.
-  Q. Zhang, R. Cao, Y. Wu, and S.-C. Zhu. Interpreting cnn knowledge via an explanatory graph. In arXiv:1708.01785, 2017.
-  Q. Zhang, R. Cao, Y. N. Wu, and S.-C. Zhu. Growing interpretable part graphs on convnets via multi-shot learning. In AAAI, 2017.
-  B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Object detectors emerge in deep scene cnns. In ICRL, 2015.
-  B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning deep features for discriminative localization. In CVPR, 2016.
-  L. Zhu, Y. Chen, Y. Lu, C. Lin, and A. Yuille. Max-margin and/or graph learning for parsing the human body. In CVPR, 2008.