D-PCN: Parallel Convolutional Neural Networks for Image Recognition in Reverse Adversarial Style

by   Shiqi Yang, et al.

In this paper, a recognition framework named D-PCN using a discriminator is proposed, which can intensify the feature extracting ability of convolutional neural networks. The framework contains two parallel convolutional neural networks, and a discriminator, which is introduced from the Generative Adversarial Nets and can improve the performance of parallel networks. The two nets are devised side by side, and the discriminator takes in the features from parallel networks as input, aiming to guide the two nets to learn features of different details in a reverse adversarial style. After that, the feature maps from two nets get aggregated, then an extra overall classifier is added and will output the final prediction employing the fused features. The training strategy of the D-PCN is also introduced which ensures the utilization of the discriminator. We experiment the D-PCN with several CNN models including NIN, ResNet, ResNeXt and DenseNet using single NVIDIA TITAN Xp, on the two benchmark datasets: CIFAR-100 and downsampled ImageNet-1k, the D-PCN enhances all models on CIFAR-100 and also reinforces the performance of ResNet on downsampled ImageNet-1k explicitly. In particular, it yields state-of-the-art classification performance on CIFAR-100 with compared to relative works.



There are no comments yet.


page 1

page 3

page 4


Parallel Convolutional Networks for Image Recognition via a Discriminator

In this paper, we introduce a simple but quite effective recognition fra...

Deeply-Supervised Nets

Our proposed deeply-supervised nets (DSN) method simultaneously minimize...

Feature Fusion for Online Mutual Knowledge Distillation

We propose a learning framework named Feature Fusion Learning (FFL) that...

Bio-inspired Min-Nets Improve the Performance and Robustness of Deep Networks

Min-Nets are inspired by end-stopped cortical cells with units that outp...

AFINet: Attentive Feature Integration Networks for Image Classification

Convolutional Neural Networks (CNNs) have achieved tremendous success in...

Adversarial-Based Knowledge Distillation for Multi-Model Ensemble and Noisy Data Refinement

Generic Image recognition is a fundamental and fairly important visual p...

Convolutional Dynamic Alignment Networks for Interpretable Classifications

We introduce a new family of neural network models called Convolutional ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Since the AlexNet [1] sparked off the passion for research on convolutional neural networks (CNNs), CNNs have been improving the performance of image classification continuously. And heterogeneous successive brilliant CNN models lead this wave with compelling results, besides, state of the art of various vision tasks, such as detection [2, 3, 4] and segmentation [5, 6], is advancing rapidly leveraging the power of CNN.

A number of recent papers [7, 8, 9, 10, 11] have tried to lend insights on the interpretability of CNN. These methods focus on understanding CNN by visualizing the learned representations. An interesting conclusion [8, 9] has been drawn that CNN has the ability to localize objects without any supervision in classification task. As shown in Figure 1, we visualize the VGG16 using Grad-CAM [11]. We posit that single network may not notice all informative regions or details which leads to misclassification as exhibited in Figure 1. Based on this point, in this paper we propose a parallel networks architecture dubbed D-PCN, which coordinates parallel networks to focus on different regions and learn complementary features under the guide of a discriminator. The D-PCN is depicted in Figure 2. The final prediction is reported by the extra classifier. We adopt a training method which is modified from adversarial learning to achieve our goal.

We implement D-PCN on CIFAR-100 [12] and ImageNet32x32 [13] datasets with NIN [14], ResNet [15], WRN [16], ResNeXt [17] and DenseNet [18]. In experiments, the performance of D-PCN ascends greatly compared with single base CNN. In particular, our method has outperformed all advanced related approaches which use multiple subnetworks on CIFAR-100. We also apply Grad-CAM [11] to visualize D-PCN with VGG16 [19] on a fine-grained classification dataset, Stanford Dogs [20], and the result verifies our motivation. In addition, we introduce D-PCN into FCN [5] on PASCAL VOC 2012 segmentation task, and experiment result demonstrates that D-PCN can also improve accuracy of segmentation. And an arresting thing occurs in segmentation experiment, that is the degree of convergence for both parallel networks rises significantly.

Figure 1: Grad-CAM visualization of VGG16, which aims to distinguish diverse categories of dogs. The second row shows the class-discriminative regions localized by CNN. The cls result means whether network predicts correctly.
Figure 2: Illustration of the proposed D-PCN. A single network is divided into two parts: extractor and classifier. The parallel networks are expected to learn complementary features under supervision of discriminator. Features from two-stream networks are integrated and then fed into extra classifier. The final prediction is reported by the extra classifier.

We summarize our contributions as follows:

  • We propose the D-PCN, a simple but quite effective framework to enhance feature extracting ability of CNN, which outperforms all other related methods.

  • Two parallel networks in D-PCN focus on different regions of input respectively, that leads to more discriminative representations after features fusion.

  • We propose a novel training method resembling adversarial learning, and it’s of high efficiency to be applied for D-PCN.

2 Related Work

CNN based models occupy advanced performance in almost all computer vision areas, including classification, detection and segmentation. Many attempts have been made to design efficient CNN architecture. In early stage, the networks from AlexNet 

[1] to VGGnet [19] tend to get deeper, and thanks to skip connection, ResNet [15] can contain extreme deeper layers. WRN [16] demonstrates the fact that increasing width can improve performance too. And there also exist other innovative designed models, such as Inception [21], ResNeXt [17], DenseNet [18]

, which improve capability of CNN further. Besides, many new modules have been constructed. For example, serials of activation functions have been introduced, like PReLU 

[22] which accelerates convergence, and the interesting SeLU [23]

. Additionally, batch normalization 

[24] is used to normalize input of layers and improve performance. Some other works achieve enhancement by employing regularizer such as dropout [25] and maxout [26].

Although all these methods turn out to be very helpful, but sometimes designing new network models or activation units is of high complexity. Speculating on how to strengthen ability of existed CNN models is a feasible approach. Several works have paid attention to that, including Bilinear CNN [27], HD-CNN [28], DDN [29] and DualNet [30], all of which resort to multiple networks. HD-CNN [28] embeds CNN into two-level category hierarchy, it separates easy classes with a coarse category classifier while distinguishing different classes using a fine classifier. And DDN [29] automatically builds a network that splits the data into disjoint clusters of classes which would be handled by the subsequent expert networks.

Bilinear CNN [27] and DualNet [30] are more related to our work which all use parallel networks. But our work is distinctive from them. In Bilinear CNN [27] the parallel networks have different parameters numbers and receptive fields, and in fact Bilinear CNN is eventually implemented with a single CNN using weights sharing, while D-PCN has two identical networks. DualNet [30] is the first to focus on the cooperation of multiple CNNs. Although it shares same philosophy with us which means deploying identical parallel networks, it puts an extra classifier in the end to participate in joint training with parallel networks. The extra classifier guarantees divergence of two networks in DualNet, and final prediction is a weighted average over three classifiers. While D-PCN uses a discriminator to drive two networks to learn complementary representations, and the added extra classifier doesn’t take part in training with parallel. Moreover, the final prediction in D-PCN is reported by extra classifier alone. Besides, the motivation is different, two subnetworks in D-PCN are expected to localize distinctive regions of input, with which parallel networks can learn complementary features. A novel training strategy adapted from adversarial learning is proposed to achieve it in D-PCN.

3 D-Pcn

3.1 Motivation

Figure 3: The architecture of D-PCN based on ResNet-20. Noted that classifier of two networks and the discriminator just appear in training process.

Nowadays, neural networks are still trained with back propagation to optimize the loss function, the process is driven by losses generated at higher layers. Consequently, some distinctive information of object, which is low-level but vital to discriminate similar classes, may be dropped in the intermediate layers, or in some case it’s overwhelmed by massive useless or redundant information. And these may happen constantly in whole propagation process when loss signals flow from higher to lower layers. All in all, it is tough for a single network to extract all details of input.

As mentioned in Section 1, there are various intriguing works for visualization on CNN lately, which point out that CNN can localize related target object in a spontaneous way. We conjecture that the missing of some information in the optimizing process may lead to inaccurate localization and misclassification as shown in Figure 1. We want to utilize this characteristic of CNN to improve performance of vision tasks. Therefore we hope to find a way to compel multiple networks to focus on different regions or details of input, which implies one network can learn features omitted by others, thereby complementary representations can be learned.

Recently, Generative Adversarial Nets (GAN) [31] are in full bloom. In a GAN, the discriminator and generator are playing a max-min game which is realized by adversarial learning. This competition between them can drive both teams to improve their methods until the spurious are indistinguishable from the genuine articles. The adversarial learning can be depicted as below:

here represents discriminator and means generator.

Arguably, the discriminator is a relay through which generator acquires information from real article input, meanwhile it differentiates generated stuff and real stuff.  [31] reformulates as:

means Jensen-Shannon divergence. The generator is to minimize the divergence so as to generate indistinguishable stuff.

Inspired by all of these, we propose a parallel networks framework named D-PCN. In D-PCN, there are two identical parallel networks, a discriminator, and an extra classifier giving final prediction. The key component of D-PCN is the discriminator which can assemble parallel networks to learn features from different regions or aspects. It’s achieved by a training strategy resembling adversarial learning:


where the symbolize extractors of subnetwork 1 and 2 respectively, equal to generator in GAN. And means features learned by networks. The equation 1 is to enforce two subnetworks to learn two different features spaces. Unlike GAN, we want to maximize to increase distribution distance between two extractors, by which means subnetworks can learn different features. Details will be discussed in Section 3.3.

3.2 Architecture

The overall structure of D-PCN is illustrated in Figure 2, where there are two subnetworks with same architecture. Two subnetworks can be replaced with any present CNN model.

In order to articulate clearly, we take a D-PCN based on ResNet-20 [15] for example, which is presented in Figure 3. We separate a single network into an extractor and a classifier, corresponding to the figure. The classifier merely contains a pool layer and a fully connected (fc

) layer, the other lower layers belong to extractor. The discriminator is comprised by several convolutional layers, with batch normalization and Leaky ReLU 

[32]. Noted that in experiments sigmoid activation is added in the end of discriminator specially for D-PCN based on NIN [14].

The reasons why we choose features of higher layer to be sent to discriminator lie in two aspects. First, CNN extracts hierarchical representations from edges to almost entire object with encoded features [7], so the features in high layers are much discriminative. Discriminator which takes in these features can acquire enough information to guide training. Second, the feature size in higher layers is much small, and this will keep computational cost at bay. All other D-PCNs with various CNN models adopt similar place as division of extractor and classifier. Just as shown in Figure 3, during training procedure, extractors along with their own classifiers and discriminator will get trained in the beginning. Then features will be integrated and input to extra classifier, which will be trained solely. In inference stage, there are no discriminator and classifier in both subnetworks, and final prediction is reported by extra classifier.

For simplicity, we adopt concatenating as fusing method. After features integrated, the whole framework can obtain more discriminative representations. The extra classifier keeps same architecture as classifiers in subnetworks, but with width doubling because of concatenating. By the way, since NIN just has a pool layer in the end for prediction, we add a fc layer in extra classifier to keep proper structure.

3.3 Training Method

Figure 4: The training strategy of D-PCN. At first, parallel networks start being trained, the loss of discriminator is attached to one of them to realize different parameters of two networks. Then two subnetworks are trained jointed in accompany with discriminator. In completion of joint training, extra classifier will be trained with fused features from two extractors.

The proposed training method is crucial to coordinate parallel networks to localize diversely and learn complementary features. The process contains three steps. The discriminator in D-PCN works like a binary classifier, and it tells which network the features are from, and the loss signal it spreads to parallel networks will encourage one network to learn features omitted by another. Intuitively, discriminator is analogous to parents which ask child 2 (subnetwork2) not to touch head and back of a cat like child 1 (subnetwork1 learns some features), then child 2 is to touch back and tail of cat (learns different features), as a result two children all touch the cat and feel pleased (all learn discriminative features).

3.3.1 Training Step 1

Because it’s tough for parallel networks with same values of parameters to converge by our joint training method, we need to initialize subnetworks with different weights. Since we have a discriminator in D-PCN, we opt to make full use of it. In this stage, discriminator is initialized and fixed, and the loss value from discriminator is added to one of the subnetwork, by which means features learned by parallel networks can be discriminative and distinctive simultaneously. For subnetwork 1, the loss function is defined as:


while loss function of another is defined as:


The means cross entropy loss for classification, and

is a L2 loss from discriminator. After a few epoches, Step 1 is finished.

3.3.2 Training Step 2

Joint training on the basis of Equation 1 starts. Loss function of subnetwork 1 is defined as:


In the meantime the corresponding one of subnetwork 2 is defined as:


As for discriminator, it follows the paradigm of GAN:


Extractors in parallel networks can be regarded as counterparts of generator in GAN. In above training, ensures that learned features are discriminative, meanwhile , make sure that features from subnetworks are different. So features from two-stream nets are complementary, although duplication exists. By the way, and both can be seen as regularization to some extend.

3.3.3 Training Step 3

In this stage, we remove classifiers in two-stream networks, so does discriminator. Features from two networks get integrated and are sent to extra classifier. All extractors are fixed, and we only train extra classifier.

For classification task, is set to 1, and we find it’s sufficient to promote performance of CNN significantly although loss from discriminator will be very small.

We emphasize that discriminator receives discriminative features from subnetwork 1 and can instruct the training of subnetwork 2, although the process is operated in class level. The discriminator plays just like a kind of attention model, but it works in the form of adding loss value to subnetworks instead of weighting.

4 Experiments

In this section, we empirically demonstrate the effectiveness of D-PCN with several CNN models on various benchmark datasets and compare it with related state of the art methods. Additional experiments for visualization and segmentation are also conducted. All experiments are implemented with PyTorch

111http://pytorch.org/ on a TITAN Xp GPU.

4.1 Classification results on CIFAR-100

Method Test Accuracy
Maxout Network [26] 61.43%
Tree based priors [33] 63.15%
Network in Network [14] 64.32%
DSN [34] 65.43%
NIN+LA units [35] 65.60%
HD-CNN* [28] 67.38%
DDN [29] 68.35%
DNI, DualNet [30] 69.76%
D-PCN (ours) 71.10%
Table 1: Compared with recent related works on CIFAR-100 without data augmentation. The accuracy means the top-1 accuracy on CIFAR-100 test datasets. *-with data augmentation and 10 view testing.

The CIFAR-100 [12]

dataset consists of 100 classes and total 60000 images with 32x32 pixels each, in which there are 50000 for training and 10000 for testing. We simply apply normalization for images using means and standard deviations in three channels.

For convenience of making comparison with related works, which use NIN [14] as base model, such as HD-CNN [28], DDN [29], DualNet [30], we build a D-PCN based on NIN. We follow the setting of NIN in [29, 30], and D-PCN is trained without data augmentation. Table 1 shows performance comparisons between several works. Noted directly compared to D-PCN are [14, 28, 29, 30], which are all built on NIN. And HD-CNN actually uses cropping and 10 view testing [1] as data augmentation, but it’s a representative work using multiple subnetworks, for which reason it’s listed here. To the best of our knowledge, DualNet reports highest accuracy on CIFAR-100 without augmentation before D-PCN. Our work surpasses DualNet by 1.34%.

Furthermore, we make several elaborate comparisons between D-PCN and DualNet in order to prove the effectiveness of our work. As shown in Table 2, we list all prediction results in DualNet and D-PCN. DualNet consists of two parallel networks and an extra classifier, final prediction is given by a weighted average of all three classifiers, while ours is provided by extra classifier alone. For ResNet, DualNet takes additional data augmentation approaches, which may explain why accuracy of base network in DualNet overtakes ours. However, D-PCN still outperforms DualNet except for ResNet-20. The accuracy of two subnetworks in D-PCN already exceeds base network. It’s worth nothing that the extra classifiers achieve significant boost over base single network after features integration. These promising results may signify representations learned by parallel networks in D-PCN are indeed complementary.

DualNet [30] NIN ResNet-20* ResNet-34* ResNet-56*
base network 66.91% 69.09% 69.72% 72.81%
iter training (Extra Classifier) 69.01% 71.93% 73.06% 75.24%
iter training (classifier average) 69.51% 72.29% 73.31% 75.53%
joint finetuning (classifier average) 69.76% 72.43% 73.51% 75.57%
D-PCN NIN ResNet-20+ ResNet-34+ ResNet-56+
base network 66.63% 67.89% 68.70% 71.98%
classifier of subnet1 68.03% 68.15% 69.76% 73.44%
classifier of subnet2 67.96% 68.69% 69.94% 74.01%
extra classifier 71.10% 72.39% 74.07% 76.39%
Table 2: Comparison between DualNet and our D-PCN on CIFAR-100. Several CNN models are deployed to make comparison between DualNet and D-PCN. Here we report predictions from all classifiers in two frameworks. The top half shows the results of DualNet, in which the classifier average

means the weighted average of all three classifiers in DualNet. The bottle half shows the results of our D-PCN. * means that DualNet uses changing contrast, brightness and color shift as additional data augmentation ways, while + means our D-PCN only adopts cropping randomly with padding.

Moreover, we also evaluate other celebrated CNN models, including WRN [16], DenseNet [18] and ResNeXt [17]. The results are shown in Table 3. We can find D-PCN improve the accuracy of all these models.

D-PCN DenseNet-40 WRN-16-4 ResNeXt-29,8x64d
base network 70.03% 76.72% 81.77%
classifier of subnet1 70.33% 77.63% 81.98%
classifier of subnet2 70.10% 77.76% 82.41%
extra classifier 71.43% 80.19% 84.59%
Table 3: Accuracy of D-PCNs based on several models on CIFAR-100. All results are run by ourselves and produced with cropping randomly implemented as the only data augmentation method.

Analysis of D-PCN on ResNet and DenseNet Just as discussed in DPN [36], ResNet tends to reuse the feature and fails to explore new ones while DenseNet is able to extract new features. In short, DenseNet can learn comprehensive features as much as possible. And results in Table 2 and Table 3 reflect these characteristics, where D-PCN can bring more promotions for ResNet than DenseNet.

Compared with model ensemble

For sake of further verifying effectiveness of D-PCN, we also report results of model ensemble. Specifically, we train two CNN models independently, initialized with normal distribution or using Xavier 

[37] and Kaiming [22] initialization, and final prediction is obtained with their predictions averaged. As illustrated in Table 4, ensemble is still inferior to D-PCN. More importantly, D-PCN is orthogonal to model ensemble just like HD-CNN [28] and DaulNet [30], ensemble of D-PCNs can further improve the performance.

NIN ResNet-20 ResNeXt-29,8x64d
base network 66.63% 67.89% 81.77%
model ensemble 68.94% 70.01% 83.03%
D-PCN 71.10% 72.39% 84.59%
Table 4: Comparison between model ensemble and D-PCN on CIFAR-100. Model ensemble deploys two identical CNN models with different initialization and takes averaged prediction as final result.

Compared with doubling width We take NIN for this experiment. After doubling number of channels directly, accuracy is improved to 68.89%, still lower than 71.10% of D-PCN.

Where to feed the discriminator We take ResNet-20 for this experiment. Original D-PCN sends features from block3 to discriminator as shown in Figure 3, here we choose block2 and block1 instead. And D-PCN just gets 71.50% and 68.78% accuracy respectively, lower than 72.39% of original one. Furthermore, we try to bring features from both block2 and block3 to discriminator through a convolutional layer, and we achieve 72.89% accuracy, a little higher than 72.39% of original D-PCN.

How to aggregate features Here we experiment on NIN and WRN-16-4. We replace concatenating with sum in D-PCN, and it achieves 70.27%, 78.84% for NIN and WRN respective, lower than 71.10% and 80.19% using concatenating.

More subnetworks We take NIN for this experiment. By adjusting the loss function in Section 3.3, which is clarified in supplementary material, we can deploy three parallel networks in D-PCN. We get 71.97% accuracy, a little higher than 71.10% of original one.

4.2 Classification Results on ImageNet32x32

Downsampled ImageNet [13] has been released as a substitute for ImageNet [38] which is hard to be trained from scratch. The downsampled ImageNet has all images in original ImageNet with three kinds of resolution, i.e., 16x16, 32x32 and 64x64. Here we adopt the one with 32x32 pixels each (ImageNet32x32) as experiment dataset for testing efficiency of D-PCN on large scale dataset.

In this experiment, we investigate D-PCN with ResNet-18 [15] on ImageNet32x32. The applied ResNet-18 has same structure as ResNet for CIFAR, but numbers of channels keep pace with ResNet for ImageNet. We only shift dataset to range from 0 to 1 and then zero-center the datasets using means which are provided in ImageNet32x32. The result is shown in Table 5.

D-PCN Top1 accuracy Top5 accuracy
base ResNet-18 45.738% 59.78%
classifier of subnet1 45.732%
classifier of subnet2 45.884%
extra classifier 50.132% 69.23%
Table 5: Accuracy of D-PCN on ImageNet32x32 without data augmentation. No data augmentation method is adopted except zero-centering preprocessing.

Noted that ImageNet32x32 has huge amounts of classes and millions of images with just 32x32 pixels each, it’s quite challenging compared to original ImageNet. D-PCN attains 4.394% and 9.45% promotion in Top1 and Top5 accuracy respectively. Result of experiment for D-PCN on original ImageNet can also be found in supplementary material.

For all above experiments, since philosophy of D-PCN is to coordinate two-stream networks to learn complementary representations, it’s natural to use single CNN as baseline. And we can draw a conclusion that our work makes sense by introducing the novel framework and training strategy to compel two CNNs to learn complementary features.

4.3 Visualization

In all experiments, the loss of discriminator is becoming quite small later. We conjecture that in adversarial learning discriminator always wins, i.e., loss of the discriminator goes to very low fast. And the training rule we use in section 3.3 will make optimization of discriminator even easy. So we wonder whether discriminator takes effect and two networks really focus on different regions agreeing with our motivation.

To confirm our point of view, we apply Grad-CAM [11] to give visual explanations for base VGG16 and two networks in D-PCN on Stanford Dogs [20] dataset, which has larger resolution better for visualization. Grad-CAM is extended from CAM [9] and is applicable to a wide variety of CNN models. Stanford Dogs [20] is a fine grained classification dataset, which has 120 categories of dogs and total 20580 images, where 12000 are for training. It’s quite suitable for proving effectiveness of D-PCN since some different categories of dogs have very similar traits and are hard to be distinguished. D-PCN based on VGG16 is trained like other CNN models above. Here we select some representative pictures for visualization, as shown in Figure 5, where red zone represents class-discriminative regions for network while blue zone is on the contrary. Taking picture 4 in Figure 5 for example, original VGG only focuses on mouth of the dog, which maybe the reason of misclassification. Subnetworks localize more related areas and final result of D-PCN is correct.

And we select two images consisting of a cat, and a dog belonging to one category in Stanford Dogs, to test the trained VGG and D-PCN. Images are from internet. Visualization is shown in Figure 6. Here we list category predictions of networks. That manifests the reason why CNNs misclassify some categories of dogs maybe some features vital to distinguish similar object get omitted, since network can’t observe some aspects of object. An extreme arresting discovery is that two networks can even localize the cat with no supervision information (Please pay attention to blue zone).

From these experiments, we can see that two subnetworks not only focus on different regions, but also localize more accurately than base network, which means discriminator does play a part in D-PCN even though loss from discriminator will be very small in training. Sometimes one of parallel networks predicts wrong or both misclassify objects, but extra classifier in D-PCN gives right answer eventually. This certifies the complementarity of features from two-stream networks in D-PCN. By the way, accuracy of base VGG16 is 72.88%, and accuracy of subnetworks and extra classifier are 73.36%,73.53% and 75.86% respectively. More visualizations can be found in supplementary material which further verifies our motivation.

Figure 5: Grad-CAM visualization of VGG16 on Stanford Dogs. The correct or wrong means whether input is classified correctly by its own classifier. Last row represents results of extra classifier in D-PCN. There is no visualization for extra classifier since it takes in fused features.
Figure 6: Grad-CAM visualization of VGG16 trained on Stanford Dogs. Two pictures are from internet. On the right of dotted line is visualization of D-PCN. Below pictures are predictions, correctness of final prediction from D-PCN is in the rightmost.

4.4 Segmentation on PASCAL VOC 2012

Since D-PCN can coordinate parallel networks to learn complementary features, we think it can improve performance of other vision tasks. Here we put D-PCN into FCN [5] on PASCAL VOC 2012 semantic segmentation mission. ResNet-18 and ResNet-34 are chosen as base model. In training we set all in Section 3.3 to 0.2, and extra classifier turns into convolutional layers corresponding to FCN. Two networks are initialized with pre-trained model on ImageNet. Experiment results are shown in Table 6. D-PCN achieves 1.458% and 1.304% mIoU improvement respectively. Although improvements of testing mIoU are quite small, we found that training mIoU increases greatly. The results imply that convergence of networks rises significantly222We think it may explain why parallel networks can localize cat in Section 4.3, because subnetworks catch enough information to know what’s docg..

ResNet-18 ResNet-34
base model
training mIoU of base model 69.139% 74.259%
testing mIoU of base model 50.352% 55.335%
training mIoU of subnetwork1 85.436% 87.557%
training mIoU of subnetwork2 85.341% 87.647%
testing mIoU of subnetwork1 50.536% 56.026%
testing mIoU of subnetwork2 50.601% 56.101%
testing mIoU of D-PCN 51.810% 56.639%
Table 6: Segmentation results on PASCAL VOC 2012. We take FCN as base segmentation model. Training part of dataset is for training while validation part is for testing.

5 Conclusion

In this paper, we propose a novel framework named D-PCN to boost the performance of CNN. The parallel networks in D-PCN can learn discriminative and distinctive features via a discriminator. The fused features are more discriminative. An effective training method resembling adversarial learning is introduced. D-PCNs based on several CNN models such as NIN, ResNet, WRN, ResNeXt and DenseNet are investigated on CIFAR-100 and ImageNet32x32 datasets, and achieve promotion. In particular, it gets state-of-the-art performance on CIFAR-100 compared with related works. Additional experiments are conducted for visualization and segmentation. In the future, we will deploy D-PCN in other tasks efficiently, such as detection and segmentation, and try to transform D-PCN into a single network. We hope the philosophy of D-PCN and the discovery in this work can advance researches in several computer vision areas.


  • [1] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS. (2012)
  • [2] Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. In: NIPS. (2015)
  • [3] Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection. In: CVPR. (2016)
  • [4] Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C.: Ssd: Single shot multibox detector. In: ECCV. (2016)
  • [5] Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CVPR. (2015)
  • [6] Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. PAMI (2017)
  • [7] Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: ECCV. (2014)
  • [8] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Object detectors emerge in deep scene cnns. In: ICLR. (2015)
  • [9] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.:

    Learning deep features for discriminative localization.

    In: CVPR. (2016)
  • [10] Bau, D., Zhou, B., Khosla, A., Oliva, A., Torralba, A.: Network dissection: Quantifying interpretability of deep visual representations. In: CVPR. (2017)
  • [11] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: ICCV. (2017)
  • [12] Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. (2009)
  • [13] Chrabaszcz, P., Loshchilov, I., Hutter, F.: A downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint arXiv:1707.08819 (2017)
  • [14] Lin, M., Chen, Q., Yan, S.: Network in network. In: ICLR. (2014)
  • [15] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR. (2016)
  • [16] Zagoruyko, S., Komodakis, N.: Wide residual networks. In: BMVC. (2016)
  • [17] Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: CVPR. (2017)
  • [18] Huang, G., Liu, Z., van der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: CVPR. (2017)
  • [19] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  • [20] Khosla, A., Jayadevaprakash, N., Yao, B., Li, F.F.: Novel dataset for fine-grained image categorization: Stanford dogs. In: CVPR Workshop. (2011)
  • [21] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: CVPR. (2015)
  • [22] He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: ICCV. (2015)
  • [23] Klambauer, G., Unterthiner, T., Mayr, A., Hochreiter, S.: Self-normalizing neural networks. In: NIPS. (2017)
  • [24] Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: ICML. (2015)
  • [25] Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: A simple way to prevent neural networks from overfitting.

    The Journal of Machine Learning Research

    15(1) (2014) 1929–1958
  • [26] Goodfellow, I.J., Warde-Farley, D., Mirza, M., Courville, A., Bengio, Y.: Maxout networks. In: ICML. (2013)
  • [27] Lin, T.Y., RoyChowdhury, A., Maji, S.: Bilinear cnn models for fine-grained visual recognition. In: ICCV. (2015)
  • [28] Yan, Z., Zhang, H., Piramuthu, R., Jagadeesh, V., DeCoste, D., Di, W., Yu, Y.: Hd-cnn: hierarchical deep convolutional neural networks for large scale visual recognition. In: ICCV. (2015)
  • [29] Murthy, V.N., Singh, V., Chen, T., Manmatha, R., Comaniciu, D.: Deep decision network for multi-class image classification. In: CVPR. (2016)
  • [30] Saihui Hou, X.L., Wang, Z.: Dualnet: Learn complementary features for image recognition. In: ICCV. (2017)
  • [31] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: NIPS. (2014)
  • [32] Maas, A.L., Hannun, A.Y., Ng, A.Y.: Rectifier nonlinearities improve neural network acoustic models. In: ICML Workshop. (2013)
  • [33] Srivastava, N., Salakhutdinov, R.R.:

    Discriminative transfer learning with tree-based priors.

    In: NIPS. (2013)
  • [34] Lee, C.Y., Xie, S., Gallagher, P., Zhang, Z., Tu, Z.: Deeply-supervised nets.

    In: Artificial Intelligence and Statistics. (2015) 562–570

  • [35] Agostinelli, F., Hoffman, M., Sadowski, P., Baldi, P.: Learning activation functions to improve deep neural networks. arXiv preprint arXiv:1412.6830 (2014)
  • [36] Chen, Y., Li, J., Xiao, H., Jin, X., Yan, S., Feng, J.: Dual path networks. In: NIPS. (2017)
  • [37] Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: AISTATS. (2010)
  • [38] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: CVPR. (2009)

Appendix 0.A More subnetworks

By adjusting the loss function in Section 3.3, we can deploy three parallel networks in D-PCN.

Loss functions in training step 2 are defined as following.


Loss functions for subnetwork 2&3 in training step 1 stay the same as in step 2. And is set to 1.

DualNet [30] D-PCN
base NIN-ImageNet 59.15% 58.94%
Top 1 Accuracy 60.44% 60.97%
Table 7: Top 1 accuracy on ILSVRC-2012 ImageNet. Both DualNet and D-PCN are based on NIN-ImageNet. The Top 1 accuracy means accuracy of final prediction in DualNet and D-PCN on validation set.

Appendix 0.B Classification Results on ImageNet

To further prove efficiency of D-PCN, we train a D-PCN based on NIN with ImageNet. The structure of NIN-ImageNet stays the same as in supplementary material of DualNet [30]. And we modify the architecture of discriminator to adapt to feature size. Cropping randomly is the only adopted data augmentation method, just like DualNet. Other setting remains same as D-PCN based on NIN for CIFAR-100. As presented in Table 7, D-PCN surpasses DualNet, and gains 2.03% improvement versus base NIN-ImageNet.

Appendix 0.C Visualizations for all images through Grad-CAM

The visualizations through Grad-CAM for all images are shown as following. GB means Guided Backpropagation, and GB-CAM means GB + Grad-CAM which is achieved by fusing GB and Grad-CAM. According to 

[11], GB highlights all contributing features and GB-CAM can identify important features like stripes, pointy ears and eyes. Difference can be found between base VGG network and D-PCN in Figure 7 and Figure 8, which demonstrates that features from two subnetworks in D-PCN are indeed diverse.

Figure 7: Grad-CAM visualization of VGG16 trained on Stanford Dogs. Best viewed in color/screen.
Figure 8: Grad-CAM visualization of VGG16 trained on Stanford Dogs. Best viewed in color/screen.