Large neural networks have recently demonstrated impressive performance on a range of speech and vision tasks. However, the size of these models can make their deployment at test time problematic. For example, mobile computing platforms are limited in their CPU speed, memory and battery life. At the other end of the spectrum, Internet-scale deployment of these models requires thousands of servers to process the 100’s of millions of images per day. The electrical and cooling costs of these servers is significant. Training large neural networks can take weeks, or even months. This hinders research and consequently there have been extensive efforts devoted to speeding up training procedure. However, there are relatively few efforts aimed at improving thetest-time performance of the models.
We consider convolutional neural networks (CNNs) used for computer vision tasks, since they are large and widely used in commercial applications. These networks typically require a huge number of parameters (in ) to produce state-of-the-art results. While these networks tend to be hugely over parameterized , this redundancy seems necessary in order to overcome a highly non-convex optimization . As a byproduct, the resulting network wastes computing resources. In this paper we show that this redundancy can be exploited with linear compression techniques, resulting in significant speedups for the evaluation of trained large scale networks, with minimal compromise to performance.
We follow a relatively simple strategy: we start by compressing each convolutional layer by finding an appropriate low-rank approximation, and then we fine-tune the upper layers until the prediction performance is restored. We consider several elementary tensor decompositions based on singular value decompositions, as well as filter clustering methods to take advantage of similarities between learned features.
Our main contributions are the following: (1) We present a collection of generic methods to exploit the redundancy inherent in deep CNNs. (2) We report experiments on state-of-the-art Imagenet CNNs, showing empirical speedups on convolutional layers by a factor ofand a reduction of parameters in fully connected layers by a factor of .
Notation: Convolution weights can be described as a -dimensional tensor: . is the number of number of input channels, and are the spatial dimensions of the kernel, and
is the target number of feature maps. It is common for the first convolutional layer to have a stride associated with the kernel which we denote by. Let denote an input signal where is the number of input maps, and and are the spatial dimensions of the maps. The target value, , of a generic convolutional layer, with , for a particular output feature, , and spatial location, , is
If is a tensor, denotes its operator norm, and denotes its Frobenius norm.
2 Related Work
Vanhoucke et al. 
explored the properties of CPUs to speed up execution. They present many solutions specific to Intel and AMD CPUs, however some of their techniques are general enough to be used for any type of processor. They describe how to align memory, and use SIMD operations (vectorized operations on CPU) to boost the efficiency of matrix multiplication. Additionally, they propose the linear quantization of the network weights and input. This involves representing weights as 8-bit integers (range), rather than 32-bit floats. This approximation is similar in spirit to our approach, but differs in that it is applied to each weight element independently. By contrast, our approximation approach models the structure within each filter. Potentially, the two approaches could be used in conjunction.
The most expensive operations in CNNs are the convolutions in the first few layers. The complexity of this operation is linear in the area of the receptive field of the filters, which is relatively large for these layers. However, Mathieu et al.  have shown that convolution can be efficiently computed in Fourier domain, where it becomes element-wise multiplication (and there is no cost associated with size of receptive field). They report a forward-pass speed up of around for convolution layers in state-of-the-art models. Importantly, the FFT method can be used jointly with most of the techniques presented in this paper.
The use of low-rank approximations in our approach is inspired by work of Denil et al.  who demonstrate the redundancies in neural network parameters. They show that the weights within a layer can be accurately predicted from a small (e.g. ) subset of them. This indicates that neural networks are heavily over-parametrized. All the methods presented here focus on exploiting the linear structure of this over-parametrization.
Finally, a recent preprint  also exploits low-rank decompositions of convolutional tensors to speed up the evaluation of CNNs, applied to scene text character recognition. This work was developed simultaneously with ours, and provides further evidence that such techniques can be applied to a variety of architectures and tasks. Out work differs in several ways. First, we consider a significantly larger model. This makes it more challenging to compute efficient approximations since there are more layers to propagate through and thus a greater opportunity for error to accumulate. Second, we present different compression techniques for the hidden convolutional layers and provide a method of compressing the first convolutional layer. Finally, we present GPU results in addition to CPU results.
3 Convolutional Tensor Compression
In this section we describe techniques for compressing 4 dimensional convolutional weight tensors and fully connected weight matrices into a representation that permits efficient computation and storage. Section 3.1 describes how to construct a good approximation criteria. Section 3.2 describes techniques for low-rank tensor approximations. Sections 3.3 and 3.4 describe how to apply these techniques to approximate weights of a convolutional neural network.
3.1 Approximation Metric
Our goal is to find an approximation, , of a convolutional tensor that facilitates more efficient computation while maintaining the prediction performance of the network. A natural choice for an approximation criterion is to minimize . This criterion yields efficient compression schemes using elementary linear algebra, and also controls the operator norm of each linear convolutional layer. However, this criterion assumes that all directions in the space of weights equally affect prediction performance. We now present two methods of improving this criterion while keeping the same efficient approximation algorithms.
Mahalanobis distance metric: The first distance metric we propose seeks to emphasize coordinates more prone to produce prediction errors over coordinates whose effect is less harmful for the overall system. We can obtain such measurements as follows. Let denote the set of all parameters of the -layer network, and let
denote the output after the softmax layer of input image. We consider a given input training set with known labels . For each pair , we compute the forward propagation pass , and define as the indices of the largest values of different from . Then, for a given layer , we compute
where is the dirac distribution centered at . In other words, for each input we back-propagate the difference between the current prediction and the “most dangerous” mistakes.
The Mahalanobis distance is defined from the covariance of : where is the vector containing all the coordinates of , and is the covariance of . We do not report results using this metric, since it requires inverting a matrix of size equal to the number of parameters, which can be prohibitively expensive in large networks. Instead we use an approximation that considers only the diagonal of the covariance matrix. In particular, we propose the following, approximate, Mahalanobis distance metric:
where the sum runs over the tensor coordinates. Since (2) is a reweighted Euclidiean metric, we can simply compute , where denotes element-wise multiplication, then compute the approximation on using the standard norm, and finally output
Data covariance distance metric: One can view the Frobenius norm of as Another alternative, similar to the one considered in , is to replace the isotropic covariance assumption by the empirical covariance of the input of the layer. If is a convolutional layer, and
is the empirical estimate of the input data covariance, it can be efficiently computed as
where is the matrix obtained by folding the first three dimensions of .As opposed to , this approach adapts to the input distribution without the need to iterate through the data.
3.2 Low-rank Tensor Approximations
3.2.1 Matrix Decomposition
-tensors which can be linearly compressed using the Singular Value Decomposition. Ifis a real matrix, the SVD is defined as , where . is a diagonal matrix with the singular values on the diagonal, and , are orthogonal matrices. If the singular values of decay rapidly, can be well approximated by keeping only the largest entries of , resulting in the approximation , where Then, for , the approximation error satisfies and thus is controlled by the decay along the diagonal of . Now the computation can be done in , which, for sufficiently small is significantly smaller than .
3.2.2 Higher Order Tensor Approximations
SVD can be used to approximate a tensor by first folding all but two dimensions together to convert it into a -tensor, and then considering the SVD of the resulting matrix. For example, we can approximate as . can be compressed even further by applying SVD to . We refer to this approximation as the SVD decomposition and use and to denote the rank used in the first and second application of SVD respectively.
Alternatively, we can approximate a 3-tensor, , by a rank 1 3-tensor by finding a decomposition that minimizes
where , , and denotes the outer product operation. Problem (4) is solved efficiently by performing alternate least squares on , and respectively, although more efficient algorithms can also be considered .
This easily extends to a rank approximation using a greedy algorithm: Given a tensor , we compute using (4), and we update . Repeating this operation times results in
We refer to this approximation as the outer product decomposition and use to denote the rank of the approximation.
3.3 Monochromatic Convolution Approximation
Let denote the weights of the first convolutional layer of a trained network. We found that the color components of trained CNNs tend to have low dimensional structure. In particular, the weights can be well approximated by projecting the color dimension down to a 1D subspace. The low-dimensional structure of the weights is illustrated in Figure 2.
The monochromatic approximation exploits this structure and is computed as follows. First, for every output feature, , we consider consider the matrix , where the spatial dimensions of the filter corresponding to the output feature have been combined, and find the SVD, , where , and . We then take the rank approximation of , where . We can further exploit the regularity in the weights by sharing the color component basis between different output features. We do this by clustering the left singular vectors, , of each output feature into clusters, for . We constrain the clusters to be of equal size as discussed in section 3.4. Then, for each of the output features, , that is assigned to cluster , we can approximate with where is the cluster center for cluster and and are as before.
3.4 Biclustering Approximations
We exploit the redundancy within the 4-D weight tensors in the higher convolutional layers by clustering the filters, such that each cluster can be accurately approximated by a low-rank factorization. We start by clustering the rows of , which results in clusters . Then we cluster the columns of , producing clusters . These two operations break the original weight tensor into sub-tensors as shown in Figure 1. Each sub-tensor contains similar elements, and thus is easier to fit with a low-rank approximation.
In order to exploit the parallelism inherent in CPU and GPU architectures it is useful to constrain clusters to be of equal sizes. We therefore perform the biclustering operations (or clustering for monochromatic filters in Section 3.3) using a modified version of the -means algorithm which balances the cluster count at each iteration. It is implemented with the Floyd algorithm, by modifying the Euclidean distance with a subspace projection distance.
After the input and output clusters have been obtained, we find a low-rank approximation of each sub-tensor using either the SVD decomposition or the outer product decomposition as described in Section 3.2.2. We concatenate the and spatial dimensions of the sub-tensors so that the decomposition is applied to the 3-tensor,
. While we could look for a separable approximation along the spatial dimensions as well, we found the resulting gain to be minimal. Using these approximations, the target output can be computed with significantly fewer operations. The number of operations required is a function the number of input clusters,, the output clusters and the rank of the sub-tensor approximations ( for the SVD decomposition; for the outer product decomposition. The number of operations required for each approximation is described in Table 1.
|Approximation technique||Number of operations|
|Biclustering + outer product decomposition|
|Biclustering + SVD|
Many of the approximation techniques presented here can efficiently compress the weights of a CNN with negligible degradation of classification performance provided the approximation is not too harsh. Alternatively, one can use a harsher approximation that gives greater speedup gains but hurts the performance of the network. In this case, the approximated layer and all those below it can be fixed and the upper layers can be fine-tuned until the original performance is restored.
We use the 15 layer convolutional architecture of 
, trained on the ImageNet 2012 dataset. The network contains 4 convolutional layers, 3 fully connected layers and a softmax output layer. We evaluated the network on both CPU and GPU platforms. All measurements of prediction performance are with respect to the 20K validation images from the ImageNet12 dataset.
We present results showing the performance of the approximations described in Section 3 in terms of prediction accuracy, speedup gains and reduction in memory overhead. All of our fine-tuning results were achieved by training with less than 2 passes using the ImageNet12 training dataset. Unless stated otherwise, classification numbers refer to those of fine-tuned models.
The majority of forward propagation time is spent on the first two convolutional layers (see Supplementary Material for breakdown of time across all layers). Because of this, we restrict our attention to the first and second convolutional layers in our speedup experiments. However, our approximations could easily applied to convolutions in upper layers as well.
We implemented several CPU and GPU approximation routines in an effort to achieve empirical speedups. Both the baseline and approximation CPU code is implemented in C++ using Eigen3 library  compiled with Intel MKL. We also use Intel’s implementation of openmp and multithreading. The baseline gives comparable performance to highly optimized MATLAB convolution routines and all of our CPU speedup results are computed relative to this. We used Alex Krizhevsky’s CUDA convolution routines 111https://code.google.com/p/cuda-convnet/ as a baseline for GPU comparisons. The approximation versions are written in CUDA. All GPU code was run on a standard nVidia Titan card.
We have found that in practice it is often difficult to achieve speedups close to the theoretical gains based on the number of arithmetic operations (see Supplementary Material for discussion of theoretical gains). Moreover, different computer architectures and CNN architectures afford different optimization strategies making most implementations highly specific. However, regardless of implementation details, all of the approximations we present reduce both the number of operations and number of weights required to compute the output by at least a factor of two, often more.
4.1.1 First Layer
The first convolutional layer has 3 input channels, 96 output channels and 7x7 filters. We approximated the weights in this layer using the monochromatic approximation described in Section 3.3. The monochromatic approximation works well if the color components span a small number of one dimensional subspaces. Figure 2 illustrates the effect of the monochromatic approximation on the first layer filters.
The only parameter in the approximation is , the number of color channels used for the intermediate representation. As expected, the network performance begins to degrade as decreases. The number of floating point operations required to compute the output of the monochromatic convolution is reduced by a factor of , with the larger gain resulting for small . Figure 3 shows the empirical speedups we achieved on CPU and GPU and the corresponding network performance for various numbers of colors used in the monochromatic approximation. Our CPU and GPU implementations achieve empirical speedups of relative to the baseline with less than 1% drop in classification performance.
4.1.2 Second Layer
The second convolutional layer has 96 input channels, 256 output channels and 5x5 filters. We approximated the weights using the techniques described in Section 3.4. We explored various configurations of the approximations by varying the number of input clusters , the number of output clusters and the rank of the approximation (denoted by and for the SVD decomposition and for the outer product decomposition).
Figure 4 shows our empirical speedups on CPU and GPU and the corresponding network performance for various approximation configurations. For the CPU implementation we used the biclustering with SVD approximation. For the GPU implementation we using the biclustering with outer product decomposition approximation. We achieved promising results and present speedups of relative to the baseline with less than a 1% drop in performance.
4.2 Combining approximations
The approximations can also be cascaded to provide greater speedups. The procedure is as follows. Compress the first convolutional layer weights and then fine-tune all the layers above until performance is restored. Next, compress the second convolutional layer weights that result from the fine-tuning. Fine-tune all the layers above until performance is restored and then continue the process.
We applied this procedure to the first two convolutional layers. Using the monochromatic approximation with 6 colors for the first layer and the biclustering with outer product decomposition approximation for the second layer () and fine-tuning with a single pass through the training set we are able to keep accuracy within 1% of the original model. This procedure could be applied to each convolutional layer, in this sequential manner, to achieve overall speedups much greater than any individual layer can provide. A more comprehensive summary of these results can be found in the Supplementary Material.
4.3 Reduction in memory overhead
In many commercial applications memory conservation and storage are a central concern. This mainly applies to embedded systems (e.g. smartphones), where available memory is limited, and users are reluctant to download large files. In these cases, being able to compress the neural network is crucial for the viability of the product.
In addition to requiring fewer operations, our approximations require significantly fewer parameters when compared to the original model. Since the majority of parameters come from the fully connected layers, we include these layers in our analysis of memory overhead. We compress the fully connected layers using standard SVD as described in 3.2.2, using to denote the rank of the approximation.
shows the number of parameters for various approximation methods as a function of hyperparameters for the approximation techniques. The table also shows the empirical reduction of parameters and the corresponding network performance for specific instantiations of the approximation parameters.
|Approximation method||Number of parameters||Approximation||Reduction||Increase|
|hyperparameters||in weights||in error|
|Conv layer 1: Monochromatic||0.43%|
|Conv layer 2: Biclustering||; ;||5.3||0.68%|
|+ outer product decomposition|
|Conv layer 2: Biclustering + SVD||; ;||0.9%|
|FC layer 1: Matrix SVD||0.8394%|
|FC layer 2: Matrix SVD||0.19%|
|FC layer 3: Matrix SVD||0.67%|
In this paper we have presented techniques that can speed up the bottleneck convolution operations in the first layers of a CNN by a factor , with negligible loss of performance. We also show that our methods reduce the memory footprint of weights in the first two layers by factor of and the fully connected layers by a factor of . Since the vast majority of weights reside in the fully connected layers, compressing only these layers translate into a significant savings, which would facilitate mobile deployment of convolutional networks. These techniques are orthogonal to other approaches for efficient evaluation, such as quantization or working in the Fourier domain. Hence, they can potentially be used together to obtain further gains.
An interesting avenue of research to explore in further work is the ability of these techniques to aid in regularization either during or post training. The low-rank projections effectively decrease number of learnable parameters, suggesting that they might improve generalization ability. The regularization potential of the low-rank approximations is further motivated by two observations. The first is that the approximated filters for the first conolutional layer appear to be cleaned up versions of the original filters. Additionally, we noticed that we sporadically achieve better test error with some of the more conservative approximations.
-  Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., LeCun, Y.: Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:1312.6229 (2013)
-  Denil, M., Shakibi, B., Dinh, L., Ranzato, M., de Freitas, N.: Predicting parameters in deep learning. arXiv preprint arXiv:1306.0543 (2013)
-  Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.R.: Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580 (2012)
Vanhoucke, V., Senior, A., Mao, M.Z.:
Improving the speed of neural networks on cpus.
In: Proc. Deep Learning and Unsupervised Feature Learning NIPS Workshop. (2011)
-  Mathieu, M., Henaff, M., LeCun, Y.: Fast training of convolutional networks through ffts. arXiv preprint arXiv:1312.5851 (2013)
-  Jaderberg, M., Vedaldi, Andrea, Zisserman, A.: Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv:1405.3866 (2014)
-  Zhang, T., Golub, G.H.: Rank-one approximation to high order tensors. SIAM J. Matrix Anal. Appl. 23(2) (February 2001) 534–550
-  Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional neural networks. arXiv preprint arXiv:1311.2901 (2013)
-  Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: A Large-Scale Hierarchical Image Database. In: CVPR09. (2009)
-  Guennebaud, G., Jacob, B., et al.: Eigen v3. http://eigen.tuxfamily.org (2010)
-  Zeiler, M.D., Taylor, G.W., Fergus, R.: Adaptive deconvolutional networks for mid and high level feature learning. In: Computer Vision (ICCV), 2011 IEEE International Conference on, IEEE (2011) 2018–2025
-  Le, Q.V., Ngiam, J., Chen, Z., Chia, D., Koh, P.W., Ng, A.Y.: Tiled convolutional neural networks. In: Advances in Neural Information Processing Systems. (2010)
-  Le, Q.V., Ranzato, M., Monga, R., Devin, M., Chen, K., Corrado, G.S., Dean, J., Ng, A.Y.: Building high-level features using large scale unsupervised learning. arXiv preprint arXiv:1112.6209 (2011)
-  Lowe, D.G.: Object recognition from local scale-invariant features. In: Computer vision, 1999. The proceedings of the seventh IEEE international conference on. Volume 2., Ieee (1999) 1150–1157
-  Krizhevsky, A., Sutskever, I., Hinton, G.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems 25. (2012) 1106–1114
Appendix A Forward propagation time breakdown
Table 3 shows the time breakdown of forward propagation for each layer in the CNN architecture we explored. Close to 90% of the time is spent on convolutional layers, and within these layers the majority of time is spent on the first two.
|Layer||Time per batch (sec)||Fraction|
|Layer||Time per batch (sec)||Fraction|
Appendix B Theoretical speedups
We can measure the theoretically achievable speedups for a particular approximation in term of the number of floating point operations required to compute the target output. While it is unlikely that any implementation would achieve speedups equal to the theoretically optimal level, the number of necessary floating point operations still provides an informative upper bound on the gains.
Table 4 shows the theoretical speedup of the monochromatic approximation. The majority of the operations result from the convolution part of the computation. In comparison, the number of operations required for the color transformation is negligible. Thus, the theoretically achievable speedup decreases only slightly as the number of color components used is increased.
Figure 5 plots the theoretically achievable speedups against the drop in classification performance for various configurations of the biclustering with outer product decomposition technique. For a given setting of input and output clusters numbers, the performance tends to degrade as the rank is decreased.
|Number of colors||Increase in test error||Theoretical speedup|
Appendix C Combined results
We used the monochromatic approximation with 6 colors for the first layer. Table 5 summarizes the results after fine-tuning for 1 pass through the ImageNet12 training data using a variety of second layer approximations.
|Layer 2||Increase in error|
|+ outer product decomposition|
|+ outer product decomposition|
|Biclustering + SVD||1.2%|
|Biclustering + SVD||1.4%|