Deep convolutional neural networks have achieved great successes on many computer vision tasks, such as object classification, detection and segmentation  . ‘Depth’ played a significant role while neural networks are achieving their successes. From AlexNet to VGGNet and GoogLeNet, their performances on various computer vision tasks are boosting as network’s depth is increasing.
Experiments have shown if we simply stack layers without changing network’s structure, its performance would get worse otherwise. Because gradients of network’s parameters will vanish as depth is increasing. To settle this problem, He proposed ResNet, which introduced a residual learning framework by adding identity-mapping shortcuts. ResNet extended its depth up to over 100 layers and achieved state-of-art performances in many computer vision tasks. However, when ResNet is getting deeper(e.g. over 1000), it will suffer from the overfitting problem.
trained shallower subnetworks by randomly dropping residual modules(while retaining shortcut connections). The vanishing-gradient problem has been alleviated since only shallower networks are trained in the training phase. This training procedure can extend depth of networks to over 1000 layers(e.g. 1202 layers) and the performance on image classification has been further improved.
Zagoruyko improved ResNet from another aspect. He introduced a wider(more channels in convolution layers) and shallower ResNet variant. The performance of wide ResNet with only 16 layers exceeds that of original ResNet with over 1000 layers. Another benefit brought by wide ResNet is the training is very fast since it can take advantage of the parallel of GPU computing.
By gradually increasing width of neural networks, Han presented deep pyramidal residual Networks. For the original ResNet, width only doubled after downsampling happened. For example, there are modules in original ResNet: , , and . Width for each module are , , and . Within every module, dimensions are all the same. For pyramidal residual networks, width for each residual unit are always increasing no matter they are in the same module or not. Experiments shown pyramidal residual networks had superior generalization ability compared to the original residual networks. So except increasing the depth, properly increasing width is also another way to boost network’s performance.
Besides increasing depth or width, increasing number of shortcut connections is another effective way of improving network’s performance. It can gain network’s performance from two aspects. 1)It shortens the distance between input and output and thus alleviates the vanishing-gradient problem with shorter forward flows. Highway networks and ResNet proposed different ways of shortcut connections, both of which made training easier. 2)Shortcut connections can take advantage of multi-scale feature maps, which can improve performances on various computer vision tasks.
Huangtakes this idea to the extreme. He proposed DenseNet, for the layer of which, it takes all previous layers as its input(connections of this layer is ). By this kind of network structure design, it not only alleviates vanishing-gradient problem, but also achieves better feature reuse. DenseNet achieves superior performance on datasets of CIFAR-10,CIFAR-100 and SVHN. However, it has its own disadvantages. There are total connections for a L-layer DenseNet. The excessive connections not only decrease networks’ computation-efficiency and parameter-efficiency, but also make networks more prone to overfitting. As we can see from upper of Fig.1, When I modify connections of a 40-layer DenseNet, the test error rates on CIFAR10 first decrease and then increase as connection is increasing. When the connections is , the error rate reaches the lowest, . However, as we can see from bottom of Fig.1, error rates on the training datasets is decreasing as connections is increasing.
To settle this problem, we proposed a method to sparsify DenseNet. Zeiler found out that for a deep neural network, shallower layers can learn concrete features, whereas deeper layers can learn abstract features. Based on this observation, we can drop connections from middle layers and reduce connections for each layer from O(n) to O(1). So total connections of the sparsified DenseNet is . As we can see in Fig. 2, left is a small part of DenseNet, right is a small part of SparseNet. the dotted line are dropped connections. So our idea for sparsifing is simply dropping connections from middle layers and only retaining the nearest and farthest connections. And then we can extend the network to deeper or wider, which would result in better performance. As we can see in Fig. 2, while keeping the overall parameters unchanged, by dropping some connections and then extend network’s width or depth, the performance of networks are getting better.
Beside changing networks’ depth, width or shortcut connections to boost model’s performance, we can also borrow our knowledge about human visual processing mechanism. The most significant feature of human visual system lies in its attention mechanism. When we skim images, we can automatically focus on important regions, and then devote more attentional resources to those regions. Recently some researchers on computer vision are enlightened by attention mechanism of human visual system. They designed mechanisms which can firstly select most significant regions in an image(e.g. foreground regions for object segmentation), and then pay more attention to those regions. Attention mechanism has made progresses on various computer vision tasks, such as image classification,image segmentation
, human pose estimation and so on. Recently, Hu took advantage of attention mechanism from another perspective, he put different amounts of ‘attentional resources’ to different channels of feature maps. To be specific, he increases weights on channels which have informative features and decreases weights on channels which have less useful features. He proposed SE module, which can calibrate feature responses adaptively for different channels in cost of slightly more computation and parameters. The SE module has been proved to be effective for ResNet,Inception and Inception_ResNet
. However, the improvement is ignorable when it applied to our SparseNet. To settle this problem, we present a new attention mechanism. Its structure is shown in Fig 4. It consists of one global average pooling layer and two convolution modules(includes convolution layer, ReLU layer and batch normalization layer). Borrowing idea of shortcut connections, outputs of both global average pooling layer and the first convolution module are taken as input to the second convolution module. And then outputs of the second convolution module are used to calibrate the original network’s output.
There are two contributions in our paper:
1) We present an effective way to sparsify DenseNet, which can improve network’s performance by simultaneously increasing depth, width and shortcut connections of networks. Besides,
2) we also proposed an attention mechanism, which can further boost network’s performance.
2 Related work
2.1 Convolutional neural networks
Since 2012, neural networks, as a new way of constructing models, have made big steps in various computer vision regions. AlexNet
, which consists of 8 layers, won the image classification championship of ILSVRC-2012. It reduced error rate on ImageNet dataset from(best performance in 2011) to . In 2014, when VGGNet and Inception Net were introduced, depth of networks had been easily extended to 20 layers and the accuracy of image classification also improved a lot. As network goes deeper, simply stacking layers would degrade its performance. To solve the problem, He introduced ResNet, which learns Residual function ,instead of target function directly. ResNet can be extended to over 100 layers and the performance can be further improved.
Many researches have been made on ResNet variants. He changed the conventional ”post-activation” of the weight layers to ”pre-activation”. To be specific, he put BN layer and ReLU layer before Conv layer. As the result turned out, this identity-mapping change made training easier and thus the performance of networks better. Han introduced Deep Pyramidal Residual Networks, which increase width gradually layer by layer and rearrange the convolution module. Experiments showed their network architecture has superior generalization ability compared to original ResNet.
Targ proposed ResNet in ResNet(RiR), which changed convolution module to a small deep dual-stream architecture. RiR makes network generalize between residual stream which is similar to a residual block and transient stream which is a standard convolutional layer. Huang
constructs a very deep ResNet. By randomly dropping some residual modules with probability, they can train different shallower subnetworks in the training phase. In the testing phase, they use the whole deep network, whereas recalibrated every residual module with the survival probability . In this way, ResNet can be expended to over 1200 layers. Zagoruyko introduced a ResNet variant with wider width and shallower depth, named WRN(Wide Residual Networks), which can improve ResNet’s performance further. Huang presented DenseNet with layers connected to its all previous layers. With this kind of network design, it not only accomplishes feature reuse, but also alleviates the vanishing-gradient problem.
2.2 Attention mechanism
Attention mechanism has achieved many progresses in areas such as machine translation. Recently attention mechanism is playing a significant role in various computer vision tasks. Harleylearned weights of pixels in multiple scales using attention mechanism, and calculating the weighted average value as the final result of segmentation. Chu
improve human pose estimation using multi-context attention module. They use holistic attention model to get global consistency information of human body; while using body part attention module to get detailed information for each human part. Wang proposed a residual attention network for image classification, which achieved state-of-art performance on CIFAR dataset. By attention residual learning, they can easily extend their networks to hundreds of layers. Hu proposed SENet(Squeeze-Excitation networks), which calibrate weights for different channels by explicitly modeling channel interdependencies. SENet won ILSVRC-2017 image classification championship.
We represent the input image as ,output of the layer as and each convolutional module as function . Since input to the layer is outputs of all previous layers. The formula is presented as follows:
where is the concatenation of outputs of all previous layers. DenseNet is composed of several dense blocks connected by transition layer. Normally, size of feature map decreased by for each block. For example, size of feature map for the first block is , then for the second block, for the third block. In DenseNet, number of output feature-maps for each convolution module are always the same, which is denoted by . Thus the output number of the layer is , where is the number input to the first dense block. k was referenced as .
As DenseNet goes deeper, number of input feature-maps would become excess very soon. To settle this problem, the author put convolution module(as bottleneck layer) before the convolution module. Thus, the convolutional module has changed from BN+ReLU+Conv to BN+ReLU+Conv+BN+ReLU+Conv.(And the new convolution module is counted as two layers instead of one). The normal setup for output feature-maps of bottleneck layer(Conv) is . Thus, inputs to every Conv layer is fixed to . To further improve model compactness, number of feature-maps can also be reduced in transition layer. The normal setup is number of feature-maps is reduced by . This kind of DenseNet is called DenseNet-BC.
We introduce a method to sparsify DenseNet. The basic idea is dropping connections from middle layers and preserving only the farthest and nearest connections. The formula is as followings:
denotes number of connections we will preserve(We call it ‘path’). As DenseNet does, we also use bottleneck layer and compress the model in the transition layer, the hyperparameters are set the same as that of DenseNet.
Moreover, we also make a structure optimization. In DenseNet, layer number are the same for all dense blocks. However, in our SparseNet, the layer number in each block is increasing. We will talk about the advantages of this arrangement in section 4.6.
3.3 Attention mechanism
We proposed an attention mechanism to further boost network’s performance. Structure is shown in Figure 3. Suppose the input is , the left part is a convolution module, we denoted the function as . The right part is the attention mechanism module, and denote it as . It consists of one global Pooling layer and two convolution modules. The input to the second convolution module is the concatenation of outputs of both global pooling layer and the first convolution module. Then the final result is calculated as .
To summarize, as it is shown in Fig.5, We proposed three networks. (a) is the original DenseNet; (b) is the basic SparseNet(path = , since connections to every layer is at most ); (c) is SparseNet-bc, by adding bottleneck layers and reducing number of feature-maps in transition layer; (d) is SparseNet-abc, by adding attention mechanism on SparseNet-bc. The whole framework is shown in Figure 6.
3.5 Implementation details
All our models include three sparse blocks. The layers within each sparse block are increasing. Besides bottleneck layer, all convolutional kernels are . blocks are connected with transition layer, which reduced feature map size by and feature map number by
(feature map number will remain the same for the basic SparseNet). After the last block, a global pooling layer and a softmax classifier is attached. For each network(SparseNet, SparseNet-bc, SparseNet-abc), we construct three different sizes of parameters. denoting by V1, V2, V3 and V4. For V1, the layer number for three blocks are 8,12,16; 12,18,24 for V2; 16, 24,32 for V3 and 20,30,40 for V4. other parameters are listed in table 1.
CIFAR are colored images with three channels. Their sizes are . CIFAR10 consists of 10 classes and CIFAR100 consists of 100 classes. Both are composed of 50,000 training images and 10,000 test images.
The Street View House Numbers(SVHN) are also colored images with three channels. Their sizes are . SVHN includes 73,257 training images, 531,131 additional training images and 26,032 test images. We training our model using all the training images.
. For CIFAR, the training epoch is 280. the initial learning rate is 0.1, and decreasing learning rate to 0.01,0.001,0.0002 at epoch 150, 200 and 250. For SVHN, the total epoch is 40, and decreasing to 0.01 and 0.001 at epoch 20 and 30. the batch size of both datasets are 64.
4.3 Classification Results on CIFAR and SVHN
Results on datasets of CIFAR and SVHN are shown in table 2. Compared to DenseNet, SparseNet achieves superior performances on all datasets. On CIFAR10, SparseNet decreases error rate from to . On CIFAR100, SparseNet achieves error rate of , while DenseNet achieved . On SVHN, SparseNet also achieves lower error rate( v.s. ). Furthermore, SpareNet outperforms the existing state-of-art on CIFAR10 and SVHN. Its error rates are lower than PyramidNet on CIFAR10, which achieved and DenseNet on SVHN, which achieved .
|ResNet with Stochastic Depth||1.70M||1.7M||5.23||24.58||1.75|
|FractalNet with Dropout/Drop-path||38.6M||21||5.22||23.30||2.01|
4.4 Attention mechanism
As we can see from table 2, attention mechanism can boost networks’ performance for most model sizes(V1, V2 and V3) with only increasing in parameters and increasing in inference time. We also compared our attention mechanism to SE module on SparseNet-V1. Results are shown in Fig. 7. In the whole training phase, our attention mechanism is always superior to SE module. Besides that, the effect of SE module on SparseNet-V1 is nearly neglectable.
4.5 Parameter Efficiency and Computation Efficiency of SparseNet
The results in Fig.8 indicate that SparseNet utilizes parameters more efficiently than alternative models. SparseNet-abc-v1 achieves lower test error on CIFAR10 than pre-activation ResNet of 10001 layers, while latter has times more parameters than the former one. For DenseNet-BC, the best model achieves , while SparseNet achives lower test errror() with less parameters. For the recent CondenseNet, which designed for mobile devices, Our SparseNet is still more parameter-efficient.
To analyze SparseNet’s computation, we compared FLOPs111 computed Conv2D with TensorFlow framwork
computed Conv2D with TensorFlow framwork(floating-point operations) of pre-activation ResNet, DenseNet and SparseNet. Results are shown in Fig. 8. It shows SparseNet is more computation-efficient than the other two models. Compared to the best DenseNet Model, SparseNet is faster than DenseNet.
4.6 Structure optimization
We also analyzed the effectiveness of our layer arrangement for each sparse block. We compared two kinds of block arrangements. One is the increasing arrangement: 8-12-16; the other is equal arrangement: 12-12-12. Results are listed in table 3. It shows that our increasing arrangement is superior not only in computation but also in accuracy.
|block setup||#FLOPS||test error|
5.1 Where to drop connections
In this section, we experimented different methods of reducing connections. Take SparseNet-V1(path=14) for example, we tried different ways of dropping connections:
1)14-0: only preserving the farthest 14 connections;
2)10-4: preserving the farthest 10 connections and nearest 4 connections;
3)7-7(ours): preserving the farthest 7 connections and nearest 7 connections;
4) 4-10: preserving the farthest 4 connections and nearest 10 connections;
5) 0-14: only preserving the nearest 14 connections.
As we can see from Fig. 10, different dropping connection methods resulted in different error rates. And our dropping connections method(7-7) achieves best performance. Besides 7-7, 0-14 also achieved comparable performance to our method. One possible explanation is that the method of preserving the nearest 14 connections contains as much information as method of preserving the farthest 7 connections and nearest 7 connections for SparseNet-V1.
5.2 How layers, growth rate and path influence network’s performance
We also analyzed how networks’ layer, width and shortcut connections influence network’s performance. In our experiments, we set up three layers: ( layers per block), layers( layers per block) and layers( layers per block). We set the range of growth rate(k) to be [6,26]. The parameters of all models are around 1M. So when we set the layer number and the growth rate, the number of connections(path) is also determined. The results are showed in Fig. 11. We can see that for each layer setup, all test errors are experiencing decreasing first and then increasing, resulting the optimal test error are always in the somewhere middle. For different layer setups, the lowest test error is within layers of , which is between and . The results showed that none of the three factors shouldn’t be set to be extreme. Only by increasing layers, growth rate and path synchronously, can SparseNet achieve better performance.
In this work, we proposed a method to sparsify DenseNet. After reducing shortcut connections, we can expend the network to deeper and wider. Moreover, we also introduced an attention model, which can boost networks’ performance further. Experiments showed that compared to DenseNet, our model achieved comparable performance on datasets of CIFAR and SVHN with much less parameters and much lower computation. Besides, we analyzed several ways of reducing connections and how layers, growth rate and shortcut connections influence networks’ performance. In future work, we will apply our models to other computer vision tasks, for example object detection, object segmentation, human pose estimation and so on.
-  Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems. (2012) 1097–1105
-  Girshick, R.: Fast r-cnn. arXiv preprint arXiv:1504.08083 (2015)
Long, J., Shelhamer, E., Darrell, T.:
Fully convolutional networks for semantic segmentation.
In: Proceedings of the IEEE conference on computer vision and pattern recognition. (2015) 3431–3440
-  Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
-  Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A., et al.: Going deeper with convolutions, Cvpr (2015)
-  He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. (2016) 770–778
-  Huang, G., Sun, Y., Liu, Z., Sedra, D., Weinberger, K.Q.: Deep networks with stochastic depth. In: European Conference on Computer Vision, Springer (2016) 646–661
-  Zagoruyko, S., Komodakis, N.: Wide residual networks. arXiv preprint arXiv:1605.07146 (2016)
-  Han, D., Kim, J., Kim, J.: Deep pyramidal residual networks. arXiv preprint arXiv:1610.02915 (2016)
-  Srivastava, R.K., Greff, K., Schmidhuber, J.: Training very deep networks. arXiv preprint arXiv:1507.06228 (2015)
-  Hariharan, B., Arbeláez, P., Girshick, R., Malik, J.: Hypercolumns for object segmentation and fine-grained localization. In: Proceedings of the IEEE conference on computer vision and pattern recognition. (2015) 447–456
-  Sermanet, P., Kavukcuoglu, K., Chintala, S., LeCun, Y.: Pedestrian detection with unsupervised multi-stage feature learning. In: Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on, IEEE (2013) 3626–3633
-  Yang, S., Ramanan, D.: Multi-scale recognition with dag-cnns. In: Computer Vision (ICCV), 2015 IEEE International Conference on, IEEE (2015) 1215–1223
-  Huang, G., Liu, Z., Weinberger, K.Q., van der Maaten, L.: Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. Volume 1. (2017) 3
-  Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: European conference on computer vision, Springer (2014) 818–833
-  Wang, F., Jiang, M., Qian, C., Yang, S., Li, C., Zhang, H., Wang, X., Tang, X.: Residual attention network for image classification. arXiv preprint arXiv:1704.06904 (2017)
-  Harley, A.W., Derpanis, K.G., Kokkinos, I.: Segmentation-aware convolutional networks using local attention masks. In: IEEE International Conference on Computer Vision (ICCV). Volume 2. (2017) 7
-  Chu, X., Yang, W., Ouyang, W., Ma, C., Yuille, A.L., Wang, X.: Multi-context attention for human pose estimation. arXiv preprint arXiv:1702.07432 1(2) (2017)
-  Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. arXiv preprint arXiv:1709.01507 (2017)
-  Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2016) 2818–2826
-  Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (June 2015)
-  He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks. In: European Conference on Computer Vision, Springer (2016) 630–645
-  Targ, S., Almeida, D., Lyman, K.: Resnet in resnet: generalizing residual architectures. arXiv preprint arXiv:1603.08029 (2016)
-  Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)
-  Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. (2009)
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.:
Reading digits in natural images with unsupervised feature learning.
In: NIPS workshop on deep learning and unsupervised feature learning. Volume 2011. (2011) 5
-  He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE international conference on computer vision. (2015) 1026–1034
-  Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on, IEEE (2017) 5987–5995
-  Larsson, G., Maire, M., Shakhnarovich, G.: Fractalnet: Ultra-deep neural networks without residuals. In: ICLR. (2017)
-  Huang, G., Liu, S., van der Maaten, L., Weinberger, K.Q.: Condensenet: An efficient densenet using learned group convolutions. arXiv preprint arXiv:1711.09224 (2017)