Structured Deep Neural Network Pruning by Varying Regularization Parameters

1 Introduction

In recent years, deep convolutional neural networks (CNN’s) have made remarkable success in computer vision tasks such as classification, detection and segmentation by leveraging large-scale networks learning from big amount of data. However, CNN leads to massive computation and storage consumptions, thus hindering its deployment on mobile and embedded devices. To reduce computation cost, many research works are carried out to compress the scales of CNN’s, which include designing compact network architectures, parameter quantization, matrix decomposition and parameter pruning.

Parameter pruning is a promising approach for CNN compression and acceleration, which aims at eliminating redundant model parameters with tolerable performance loss. One problem of parameter pruning is that it often produces unstructured and random connections which is hard to implement for speedup on general hardware platforms [Han et al.2016]. Even with sparse matrix kernels, the speedup is very limited [Wen et al.2015]. To solve this problem, many works focus on structured pruning which can shrink a network into a thinner one so that the implementation of the pruned network is efficient [Anwar and Sung2016, Sze et al.2017].

There are two categories for structured pruning. The first is importance-based methods, which prune weights in groups based on some established importance criteria. Another category is regularization-based methods, which add group regularization terms to the objective function and prune the weights by optimizing the objective function. In this paper, we propose a new regularization-based method to learn group sparsity which also combines the idea of importance-based methods.

Existing group regularization approaches tend to use the same regularization parameter for all weight groups in the network [Wen et al.2015, Lebedev and Lempitsky2016]

. One hidden assumption of equally assigning regularization parameters is that all weights in different groups are equally important. However, intuitively, weights with greater magnitude tend to be more important during pruning than those with smaller magnitude. Thus, we need a scheme to distinguish the importance of weights and treat them with different regularization strengths. Compared with previous works, the main novelty of our proposed method is to assign different regularization parameters to different weight groups based on their estimated importance.

One advantage of the proposed method is that it is theoretically sound. In this paper, we theoretically prove that the proposed scheme of varying regularization parameters results in zeroing out unimportant weights and finally make the network converge to a pruned one. Considering that many methods in parameter pruning are heuristic, the proposed method stands out with concrete theoretical basis.

Our method accelerates AlexNet for $4 \times$ theoretical speedup while improving its performance, and VGG-16 for $4 \times$ theoretical speedup with $0.8 %$ accuracy loss. The proposed method can also be applied to compact multi-branch networks such as ResNet-50, on which we achieves $2 \times$ speedup and only suffers $0.1 %$ accuracy loss.

2 Related Work

Parameter pruning has a long history in the development of neural networks [Reed1993], which can be categorized into importance-based and regularization-based methods. Importance-based methods prune weights in groups based on some established importance criteria. For example, Optimal Brain Damage (OBD) [LeCun et al.1990] and Optimal Brain Surgery (OBS) [Hassibi and Stork1993]

proposed an importance criteria based on the second-order derivatives of loss function derived from Taylor expansions. Deep compression

[Han et al.2015a, Han et al.2015b] pruned small-magnitude weights and obtained

9 \sim 13 \times

parameter reduction on AlexNet and VGG-16. Taylor Pruning [Molchanov et al.2017]

also derived a new importance criteria based on Taylor expansion, but they used the first-order derivatives, which is easy to access in back-propagation. Their method was shown effective to prune filters of CNN’s on transfer learning tasks.

[Li et al.2017] used

L_{1}

norm to guide one-shot filter pruning, which was proved effective on CIFAR-10 and ImageNet with VGG-16 and ResNet, but the reported speedup is very limited. Channel Pruning

[He et al.2017]

alternatively uses LASSO-regression-based channel selection and feature map reconstruction to prune filters, and achieves the state-of-the-art result on VGG-16. Among regularization-based methods,

[Weigend et al.1991] added a penalty-term to eliminate redundant weights for better generalization. Group-wise Brain Damage [Lebedev and Lempitsky2016] and Structured Sparsity Learning [Wen et al.2015] embedded Group LASSO [Yuan and Lin2006] into CNN regularization and obtained regular-shape sparsity.

Apart from parameter pruning, there are other methods for CNN model compression and acceleration, including designing compact architecture, parameter quantization and matrix decomposition. Compact architecture designing methods target designing more efficient and compact neural network architectures. For example, SqueezeNet [Iandola et al.2016] was proposed to stack compact blocks, which decreased the number of parameters by $50 \times$ less than the original AlexNet. MobileNet [Howard et al.2017] and ShuffleNet [Zhang et al.2017] leveraged new convolutional implementations to design networks for mobile applications.

Parameter quantization methods reduce CNN storage by quantizing the weights and using less representation bits. [Chen et al.2015]

proposed a hash function to group weights of each CNN layer into different hash buckets for parameter sharing. As the extreme form of quantization, binarized networks

[Courbariaux and Bengio2016, Lin et al.2016, Rastegari et al.2016] proposed to learn binary weights or activations. Quantization reduces floating computational complexity, but the actual speedup may be very related to specific hardware implementations.

Matrix decomposition decomposes large matrices into several small matrices to reduce computation. [Denton et al.2014]

showed that the weight matrix of a fully-connected layer can be compressed via truncated SVD. Tensor decomposition was proposed and obtained better compression result than SVD

[Novikov et al.2014]. Several methods based on low-rank decomposition of convolutional kernel tensors were also proposed to accelerate convolutional layers [Lebedev et al.2016, Zhang et al.2016].

3 The Proposed Method

Suppose weights of convolutional layers in a CNN form a sequence of 4-D tensors $W^{(l)} \in R^{N_{l} \times C_{l} \times M_{l} \times K_{l}}$ . Here $N_{l}$ , $C_{l}$ , $M_{l}$ and $K_{l}$ are the dimension of the $l$ th ( $1 \leq l \leq L$ ) weight tensor along the axis of filter, channel, spatial height and spatial width, respectively. Our proposed objective function for regularization can be formulated as:

E (W) = L (W) + \frac{λ}{2} R (W) + \frac{λ_{g}}{2} L \sum l = 1 R_{g} (W^{(l)})

(1)

Here $W$ represents the collection of all weights in the CNN; $L (W)$ is the loss function caused by data; $R (W)$ is non-structured regularization on every weight, which is the $L_{2}$ norm in this paper. $R_{g}$ is the structured sparsity regularization on each layer. In [Wen et al.2015, Lebedev and Lempitsky2016], the authors used the same $λ_{g}$ for all groups and adopted Group LASSO for $R_{g}$ because it can effectively zero out weights in groups, i.e., $R_{g} (ω) = \sum_{g = 1}^{G} ∥ ω^{(g)} ∥$ . In this paper, we use the squared $L_{2}$ norm for $R_{g}$ , i.e., $R_{g} (ω) = \sum_{g = 1}^{G} ∥ ω^{(g)} ∥^{2}$ and vary the regularization parameters $λ_{g}$ for different groups.

The finally learned ‘structure’ is decided by the way of splitting groups of $ω^{(g)}$ . Normally, there are filter-wise, channel-wise and shape-wise sparsity with different size of weight groups [Wen et al.2015]. The sparsity terms $R_{g}$ are represented as

$R_{g (f i l t e r)}$	$=$	$\frac{λ_{g}}{2} L \sum l = 1 ⎛ ⎝ N_{l} \sum n_{l} = 1 ∥ W_{n_{l}, :, :, :}^{(l)} ∥^{2} ⎞ ⎠$
$R_{g (s h a p e)}$	$=$	$\frac{λ_{g}}{2} L \sum l = 1 ⎛ ⎝ C_{l} \sum c_{l} = 1 M_{l} \sum m_{l} = 1 K_{l} \sum k_{l} = 1 ∥ W_{:, c_{l}, m_{l}, k_{l}}^{(l)} ∥^{2} ⎞ ⎠$
$R_{g (c h a n n e l)}$	$=$	$\frac{λ_{g}}{2} L \sum l = 1 ⎛ ⎝ C_{l} \sum c_{l} = 1 ∥ W_{:, c_{l}, :, :}^{(l)} ∥^{2} ⎞ ⎠$

In im2col implementation, weight tensors are expanded into matrices where shapes are represented as columns and filters are represented as rows. Thus, shape and filter-wise sparsity are equivalent to column and row-sparsity, as shown in Fig.1.

Figure 1: The im2col implementation of CNN is to expand tensors into matrices, so that convolutional operations are transformed to matrix multiplication. The weights at the blue squares are to be pruned. (a) Pruning a row of weight matrix is equivalent to pruning a filter in convolutional kernel. (b) Pruning a column of weight matrix is equivalent to pruning all the weights at the same position in different filters. (c) Pruning a channel is equivalent to pruning several adjacent columns in the expanded weight matrix.

3.1 Theoretical Analysis

For the proposed method, the regularization parameter $λ_{g}$ is differently assigned for different weight groups. We find that by slightly augmenting $λ_{g}$ of a weight group, and then train the network through back-propagation until the objective function, i.e. Eqn.(1), reaches the local minimum, the $L_{2}$ norm of that group will also decrease. This phenomenon is formally summarized by the following theorem.

Theorem 1

Considering the objective function

E (λ, ω) = L (ω) + \frac{λ}{2} ω^{2},

(2)

if there exists a tuple $(λ_{0}, ω_{0})$ which satisfies the following three properties:

$λ_{0} > 0$ ;
$L (ω)$ has the second derivative at $ω_{0}$ ;
$ω_{0}$ is the local minimum of function $Y_{λ_{0}} (ω) = E (λ_{0}, ω)$ ,

then there exists an $ϵ > 0$ that for any $λ_{1} \in (λ_{0}, λ_{0} + ϵ)$ , we can find an $ω_{1}$ which satisfies:

$ω_{1}$ is the local minimum of function $Y_{λ_{1}} (ω) = E (λ_{1}, ω)$ ;
$| ω_{1} | < | ω_{0} |$ .

This theorem is proved in Appendix A, which indicates that we can slightly increase the regularization parameter to compress the magnitude of weights to zero. By Eqn.(12), the magnitude of weights will be more compressed if the regularization parameter increases more.

3.2 Method Description

Theorem 1 guarantees that we can modify the $L_{1}$ norm of weight groups by increasing or decreasing their corresponding regularization parameters. Thus, we can assign different regularization parameters to weight groups based on their importance to the network. In this paper, we use the $L_{1}$ norms as the importance criterion for weight groups. Note that our method can be easily applied to other criteria such as $L_{2}$ norm and Taylor expansions.

Normalization of importance criteria is necessary because the values of $L_{1}$ norms have huge variations across different networks, layers and weight groups. Contrary to previous works, normalization is based on the ranks of weight groups in the same layer. The advantages of rank-based normalization lies in two parts – (1) Compared to other normalization methods like max/min normalization, the range of ranks is fixed from $0$ to $N_{g} - 1$ , where $N_{g}$ is the total number of weight groups in the layer; (2) For the pruning task, we need to set a pruning ratio $R$ to each layer, say, $R = 0.6$ means that we need to prune $60 %$ of weight groups which are ranked the lowest when pruning is finished. Normalization by ranks makes the pruning process controllable since it is directly towards the goal of pruning.

Specifically, we sort weight groups by their $L_{1}$ norms in ascending order. Meanwhile, to mitigate the oscillation of ranks in one training iteration, we average the rank of each group through training iterations. For a weight group, its average rank through $N$ iterations is defined as

{¯ ¯ ¯ r}_{a v g} = \frac{1}{N} N \sum n = 1 r_{n} .

(3)

Here $r_{n}$ is the rank of the $n$ th iteration. The final average rank $¯ ¯ ¯ r$ is obtained by sorting ${¯ ¯ ¯ r}_{a v g}$ of different weight groups in ascending order, making its range from $0$ to $N_{g} - 1$ .

Our aim is to assign an increment $Δ λ_{g}$ to each weight group, so that its regularization parameter $λ_{g}$ is gradually updated through the pruning process.

λ_{g}^{(n e w)} = λ_{g} + Δ λ_{g}

(4)

Following the above idea, $Δ λ_{g}$ of each group is assigned by its average rank $¯ ¯ ¯ r$ with a piecewise linear function, as shown in Eqn.(5).

Δλg(¯¯¯r)=⎧⎪ ⎪ ⎪⎨⎪ ⎪ ⎪⎩−ARNg¯¯¯r+A\emph{ if} ¯¯¯r≤RNg−ANg(1−R)−1(¯¯¯r−RNg)% \emph{ if} ¯¯¯r>RNg

(5)

Fig.2

Figure 2: The functional of $Δ λ_{g} (¯ ¯ ¯ r)$ , as defined in Eqn.(5).

depicts $Δ λ_{g} (¯ ¯ ¯ r)$ . It is seen from Fig.2 that for weight groups whose $L_{1}$ norms are small, i.e., the average ranks less than $R N_{g}$ , we need to increase their regularization parameters to further decrease their $L_{1}$ norms; and those with greater $L_{1}$ norms and rank above $R N_{g}$ , we need to decrease regularization parameters to further increase their $L_{1}$ norms. After obtaining $Δ λ_{g}$ and $λ_{g}^{(n e w)}$ by Eqn.(5) and (4), we threshold it by zero to prevent negative values of regularization:

λ_{g}^{(n e w)} = max (λ_{g} + Δ λ_{g}, 0) .

(6)

After updating $λ_{g}$ , the weights of CNN are trained through back-propagation deduced from Eqn.(1).

ω^{(n e w)} = ω - α [\frac{\partial L}{\partial ω} + (λ + λ_{g}) ω] .

(7)

After training the weights for several iterations, we recalculate $λ_{g}$ and the training process continues until convergence. Since we decrease the $L_{1}$ norms of weight groups whose ranks are less than $R N_{g}$ and increase the $L_{1}$ norms of weight groups whose ranks are greater than $R N_{g}$ , there should be $R N_{g}$ pruned weight groups at the convergence point. In Eqn.(5), $A$ is a hyper-parameter to control the speed of convergence. Greater value of $A$ results in faster convergence. Finally, we summarize the proposed algorithm in Algorithm 1.

1:Input the training set

D

, the original pre-trained model

Ω

, the non-structural regularization parameter

λ

, pruning threshold

ϵ

and target pruning ratio

R

2:Initialize

λ_{g} = 0

for each weight group.

3:Initialize the iteration number

i = 0

4:repeat

5: for each weight group in each layer do

6: Obtain

¯ ¯ ¯ r

of by Eqn.(3).

7: Update

λ_{g}

by Eqn.(5) and (6).

8: end for

9: Update weights in each weight groups through back-propagation by Eqn.(7).

10: for each weight group in each layer do

11: if the

L_{1}

norm of this group is less than

ϵ

then

12: prune this weight group permanently.

13: end if

14: end for

15:

i = i + 1

16:until The ratio of pruned weight groups reaches

R

17:

Retrain the pruned CNN for several epochs.

18:Output the pruned CNN model

Ω^{'}

Algorithm 1 The Proposed Algorithm

3.3 Discussions

The difference between Algorithm 1 and Theorem 1 is that Theorem 1 requires weights reach local minimum of the objective function (Eqn.1) before another updates of $λ_{g}$ , but in Algorithm 1, we only train the network through back-propagation for several iterations. Such a compromise is mainly attributed to time-complexity. Actually, it is time-consuming if we obtain local minimum by gradually lowering the learning rate $α$ . In addition, because objective functions of CNN’s are very complex, it is also difficult to decide whether local minimum is obtained or not. Experiments indicate that $3 \sim 5$ iterations are enough for convergence of the method.

An amazing factor of the proposed method is that it can automatically adjust search steps without any knowledge about the property of the objective function itself. By Eqn.(12), the increment of weight $d ω$ is reversely proportional to the second derivative of the objective function $L (ω)$ which is produced by CNN. Such a property makes the modification of $ω$ slower when the objective function $L (ω)$ reaches steeper areas even without knowing the exact form of $L (ω)$ , which is a nice property to make refined search of local minimum possible. We believe that the good performance of the proposed method is partly because of the automatic adjustment of search steps.

4 Experiments

We first compare fixed and varying regularization with ConvNet on CIFAR-10 dataset [Krizhevsky and Hinton2009]

. Then we evaluate the proposed method with large networks on large scale datasets. All of our experiments are conducted with Caffe. We set the weight decay factor

λ

the same with baselines, and set the hyper-parameter

A

as half of

λ

. Since we focus on acceleration in this paper, we only compress weights in convolutional layers and keep fully connected layers unchanged. Methods for comparison include Taylor Pruning (TP) [Molchanov et al.2017], Filter Pruning (FP) [Li et al.2017], Structured Sparsity Learning (SSL) [Wen et al.2015], Channel Pruning (CP) [He et al.2017] and Structured Probabilistic Pruning (SPP) [Wang et al.2017]. For all experiments, speedup is calculated by GFLOPS reduction.

4.1 Analysis with ConvNet on CIFAR-10

We firstly compare our proposed method with the Group LASSO [Yuan and Lin2006] where the regularization parameter is fixed for all weights. Group LASSO is widely applied to generate group sparsity [Lebedev and Lempitsky2016, Wen et al.2015]. The test network is ConvNet, which is a small CNN with three convolutional layers and one fully-connected layers. CIFAR-10 is a $10$ -class dataset of $60 k$ tiny images, among which $45 k$ images are used for training, $5 k$ images for validation and the other $10 k$ images for testing.

We first trained a baseline model with test accuracy $81.5 %$ . Then the proposed method is applied to learn structured sparsity, where both row sparsity and column sparsity are explored. For comparison, we employ the Group LASSO with fixed regularization parameters ( $0.04$ for row sparsity and $0.01$ for column sparsity).

Experimental results are shown in Tab.1.

Method	Row pruning		Column pruning
Method	speedup	accuracy	speedup	accuracy
SSL	$3.6 \times$	$77.3$	$3.1 \times$	$78.6$
Ours	$4.1 \times$	$79.2$	$4.6 \times$	$81.2$

Table 1: Comparison of varying and fixed regularization with ConvNet on CIFAR-10. The baseline test accuracy is

81.5 %

We can see that varying regularization consistently achieves higher speedups and accuracies than fixed regularization. Fig.3 illustrates the process that the $L_{1}$ norms of columns changes with training iterations in the conv1 layer, from which we can see that while fixed regularization gradually suppresses unimportant weights, many important weights are also unnecessarily suppressed. In varying regularization, the magnitude of some weights increases dramatically when training starts while some decreases meanwhile. Thus, the pruning process converges much faster. In addition, with varying regularization, the final weights are driven into two distinct groups – the group formed by important weights with very high $L_{1}$ norms and the group for unimportant weights with close-to-zero $L_{1}$ norms.

We also find that under similar speedup ratios, column pruning is better than row pruning in accuracy. It is because that a row typically consists of much more weights than a column and pruning rows may cause more severe side-effects for accuracy. In the following experiments, we only choose column as our sparsity group to obtain better accuracies.

Figure 3: Comparing the training process of varying regularization with that of fixed regularization. Each line illustrates the $L_{1}$ norm of a column in the conv1 layer ( $75$ columns in total).

We further compare our proposed method with more recent pruning methods, and the results are shown in Tab.2. Under different speedup ratios, our method consistently outperforms other pruning methods.

Method	Increased err. (%)
Method	$2 \times$	$4 \times$	$6 \times$
TP (our impl.)	$1.0$	$3.4$	$5.4$
FP (our impl.)	$1.6$	$3.6$	$5.0$
SSL	$3.0$	$5.8$	$7.4$
SPP	- $0.2$	$0.3$	$1.2$
Ours	- $0.5$	$0.1$	$1.0$

Table 2: The increased error of different pruning methods when accelerating ConvNet on CIFAR-10. The baseline test accuracy is

81.5 %

. Minus means the test accuracy is improved.

4.2 AlexNet on ImageNet

We apply the proposed method to AlexNet [Krizhevsky et al.2012], which is composed of $5$ convolutional layers and $3$ fully-connected layers. We download an open caffemodel from Caffe model zoo as our pre-trained model. The baseline single view top-5 accuracy on ImageNet 2012 validation dataset is $80.0 %$ . All images are rescaled to size $256 \times 256$ , then a $227 \times 227$ patch is randomly cropped from each scaled image and randomly mirrored for data augmentation. For testing, the $227 \times 227$ patches are cropped from the center of the scaled images.

Intuitively, different layers have different sensitivity to pruning, but there are few theories to quantify the redundancy of different layers in deep neural networks. Most pruning methods empirically set pruning ratio for different layers [Li et al.2017, Molchanov et al.2017, He et al.2017, Wang et al.2017]. Following these works, we empirically set the proportion of remaining columns of the five convolutional layers as $1 : 1 : 1 : 1.5 : 1.5$ . We train the baseline model with batch size $64$ for about $7$ epochs until reaching the target pruning ratio. We use a small learning rate $α = 0.0005$ and fix it during training. Then the pruned model is retrained with batch size $256$ to regain accuracy.

Experimental results are shown in Tab.3.

Method	Increased err. (%)
Method	$2 \times$	$4 \times$	$5 \times$
TP	$3.9$	$9.2$	$13.9$
FP (SPP’s impl.)	$0.6$	$4.1$	$4.7$
SSL	$1.3$	$4.3$	$5.3$
SPP	- $0.7$	$0.3$	$0.9$
Proposed	- $0.7$	- $0.2$	$0.8$

Table 3: Accelerating AlexNet on ImageNet. The baseline top-5 accuracy of the original network is

80.0 %

The proposed method and SPP are consistently better than the other three methods, and the proposed method is slightly better than SPP on average. With $4 \times$ speedup, our method can even improve the accuracy while SPP degrades the accuracy in this situation.

4.3 VGG-16 on ImageNet

We further demonstrate our method on VGG-16 [Simonyan and Zisserman2014], which has $13$ convolutional layers and $3$ fully-connected layers. We download the open caffemodel as our pre-trained model, whose single-view top-5 accuracy on ImageNet 2012 validation dataset is $89.6 %$ . Similar to AlexNet, the resized images are randomly cropped to input size $224 \times 224$ and mirrored for data augmentation in training.

Previous works [He et al.2017, Wang et al.2017] found that lower layers were more redundant in VGG-16. Therefore, the proportion of remaining ratios of low layers (conv1_x to conv3_x), middle layers (conv4_x) and high layers (conv5_x) are set to $1 : 1.5 : 2$ , the same as [Wang et al.2017]. The first and last convolutional layer (conv1_1 and conv5_3) are not pruned because both of them require very small amount of GFLOPs calculation. We first train the baseline model with batch size $64$ and with a fixed learning rate $0.0005$ . Pruning is finished after around $6$ epochs. Then the network is fine-tuned with batch size $256$ to regain accuracy.

Experimental results are shown in Tab.4. Our method is slightly better than CP and SPP, and beats TP and FP by a significant margin.

Method	Increased err. (%)
Method	$2 \times$	$4 \times$	$5 \times$
TP	$-$	$4.8$	$-$
FP (CP’s impl.)	$0.8$	$8.6$	$14.6$
CP	$0$	$1.0$	$1.7$
SPP	$0$	$0.8$	$2.0$
Ours	$0$	$0.8$	$1.5$

Table 4: Acceleration of VGG-16 on ImageNet. The values are increased single-view top-5 error on ImageNet. The baseline top-5 accuracy of the original network is

89.6 %

4.4 ResNet-50 on ImageNet

Unlike AlexNet and VGG-16, ResNet-50 is a multi-branch deep neural network, which has $53$ convolutional layers and no fully-connected layers. Open pre-trained caffemodel is adopted as our baseline, whose single view top-5 accuracy on ImageNet 2012 validation dataset is $91.2 %$ . The images are augmented the same way as in the experiment of VGG-16 (Sec.4.3).

For simplicity, we adopt the same pruning ratio $0.4$ for all $53$ convolutional layers. The training settings are similar to that of VGG-16. The pruning process stops after less than $2$ epochs before retraining.

From Tab.(5

), our method achieves much better results than two recent methods CP and SPP. This is probably because the proposed method imposes gradually regularization, making the network adapt little-by-little in the parameter space, while both the matrix decomposition in CP and direct pruning in SPP may bring much modification that a network as compact as ResNet-50 cannot endure.

Method	Increased err. (%)
CP (enhanced)	$1.4$
SPP	$0.8$
Proposed	$0.1$

Table 5: Acceleration of ResNet-50 on ImageNet. The baseline top-5 accuracy of the original network is

91.2 %

5 Conclusion

We propose structured sparsity pruning with varying regularization parameters for CNN acceleration, which assigns different regularization parameters to weight groups according to their importance to the network. Theoretical analysis guarantees the convergence of our method. The effectiveness of the proposed method is proved by comparison with state-of-the-art methods on popular CNN architectures.

Appendix A – Proof of Theorem 1

Proof: For a given $λ > 0$ , the $ω$ which is the local minimum of the function $Y_{λ} (ω) = E (λ, ω)$ should satisfy $\frac{d Y_{λ} (ω)}{d ω} = 0$ , which gives:

λ = - \frac{L^{'} (ω)}{ω} .

(8)

In this situation, we can calculate the derivative of $λ$ by using Eqn.(8):

\frac{d λ}{d ω} = \frac{L^{'} (ω) - ω L^{''} (ω)}{ω^{2}} .

(9)

Since $ω_{0}$ is the local minimum of the function $Y_{λ_{0}} (ω) = E (λ_{0}, ω)$ , it should satisfy that

{\frac{d Y_{λ_{0}} (ω)}{d ω} ∣ ∣ ∣}_{ω_{0}} = 0 % and {\frac{d^{2} Y_{λ_{0}} (ω)}{d ω^{2}} ∣ ∣ ∣ ∣}_{ω_{0}} > 0,

which yields

	$L^{'} (ω_{0}) + λ_{0} ω_{0}$	$=$	$0$		(10)
	$L^{''} (ω_{0}) + λ_{0}$	$>$	$0$		(11)

If we take Eqn.(10) into (9), we can obtain

{\frac{d λ}{d ω} ∣ ∣ ∣}_{(λ_{0}, ω_{0})} = - \frac{L^{''} (ω_{0}) + λ_{0}}{ω_{0}}

(12)

By taking Eqn.(11) into (12), we can conclude that

⎧ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎩ \begin{matrix} {\frac{d λ}{d ω} ∣ ∣ ∣}_{(λ_{0}, ω_{0})} < 0, & if ω_{0} > 0 {\frac{d λ}{d ω} ∣ ∣ ∣}_{(λ_{0}, ω_{0})} > 0, & if ω_{0} < 0 \end{matrix}

(13)

In other words, when $ω_{0}$ is greater than zero, a small increment of $λ_{0}$ will decrease the value of $ω_{0}$ ; and when $ω_{0}$ is less than zero, a small increment of $λ_{0}$ will increase the value of $ω_{0}$ . In both cases, when $λ_{0}$ increases, $| ω_{0} |$ will decrease at the new local minimum of $E (λ, ω)$ .

Thus, we finished the proof of Theorem 1.

References

[Anwar and Sung2016] S. Anwar and W. Sung. Compact deep convolutional neural networks with coarse pruning. arXiv preprint, arXiv:1610.09639, 2016.
[Chen et al.2015] W. Chen, J. T. Wilson, S. Tyree, K. Q. Weinberger, and Y. Chen. Compressing neural networks with the hashing trick. In
Proceedings of the International Conference on Machine Learning, ICML
, pages 1–10, Lille, France, 2015.
[Courbariaux and Bengio2016] M. Courbariaux and Y. Bengio. BinaryNet: Training deep neural networks with weights and activations constrained to $+ 1$ or $- 1$ . arXiv preprint, arXiv:1602.02830, 2016.
[Denton et al.2014] E. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances in Neural Information Processing Systems, NIPS, Montréal, Canada, 2014.
[Han et al.2015a] S. Han, H. Mao, and W. J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint, arXiv:1510.00149, 2015.
[Han et al.2015b] S. Han, J. Pool, and J. Tran. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, NIPS, pages 1135–1143, Montréal, Canada, 2015.
[Han et al.2016] S. Han, X. Liu, and H. Mao. EIE: efficient inference engine on compressed deep neural network. ACM Sigarch Computer Architecture News, 44(3):243–254, 2016.
[Hassibi and Stork1993] B. Hassibi and D. G. Stork. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in Information Processing Systems, NIPS, pages 164–171, Denver, CO, 1993.
[He et al.2017] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. 2017.
[Howard et al.2017] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. 2017.
[Iandola et al.2016] F. Iandola, M. Moskewicz, and K. Ashraf. SqueezeNet: Alexnet-level accuracy with 50x fewer parameters and $<$ 0.5MB model size. arXiv preprint, arXiv:1602.07360, 2016.
[Krizhevsky and Hinton2009] A. Krizhevsky and G. E. Hinton. Learning multiple layers of features from tiny images. 2009.
[Krizhevsky et al.2012] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Information Processing Systems, NIPS, pages 1097–1105, Lake Tahoe, CA, 2012.
[Lebedev and Lempitsky2016] V. Lebedev and V. Lempitsky. Fast convnets using group-wise brain damage. In
Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, CVPR
, pages 2554–2564, Las Vegas, NV, 2016.
[Lebedev et al.2016] V. Lebedev, Y. Ganin, M. Rakhuba, I. Oseledets, and V. Lempitsky. Speeding-up convolutional neural networks using fine-tuned CP-decomposition. arXiv preprint, arXiv:1510.03009, 2016.
[LeCun et al.1990] Y. LeCun, J. S. Denker, and S. A. Solla. Optimal brain damage. In Advances in Information Processing Systems, NIPS, pages 598–605, Denver, CO, 1990.
[Li et al.2017] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf. Pruning filters for efficient convnets. In International Conference on Learning Representations, ICLR, 2017.
[Lin et al.2016] Z. Lin, M. Courbariaux, R. Memisevic, and Y. Bengio. Neural networks with few multiplications. arXiv preprint, arXiv:1510.03009, 2016.
[Molchanov et al.2017] P. Molchanov, S. Tyree, and T. Karras. Pruning convolutional neural networks for resource efficient inference. In International Conference on Learning Representations, ICLR, 2017.
[Novikov et al.2014] A. Novikov, D. Podoprikhin, A. Osokin, and D. Vetrov. Tensorizing neural networks. arXiv preprint, arXiv:1509.06569, 2014.
[Rastegari et al.2016] M. Rastegari, V. Ordonez, J. Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European Conference on Computer Vision, ECCV, pages 525–542, Amsterdam, Netherland, 2016.
[Reed1993] R. Reed. Pruning algorithms – a survey. IEEE Transactions on Neural Networks, 4(5):740–747, 1993.
[Simonyan and Zisserman2014] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. Computer Science, 2014.
[Sze et al.2017] V. Sze, Y. H. Chen, T. J. Yang, and J. Emer. Efficient processing of deep neural networks: A tutorial and survey. arXiv preprint, arXiv:1703.09039, 2017.
[Wang et al.2017] Huan Wang, Qiming Zhang, Yuehai Wang, and Roland Hu. Structured probabilistic pruning for deep convolutional neural network acceleration. CoRR, abs/1709.06994, 2017.
[Weigend et al.1991] Andreas S. Weigend, David E. Rumelhart, and Barnardo A. Huberman. Generalization by weight elimination with application to forecasting. In Advances in Neural Information Processing Systems, pages 875–882, 1991.
[Wen et al.2015] W. Wen, C. Wu, and Y. Wang. Learning structured sparsity in deep neural networks. In Advances in Information Processing Systems, NIPS, pages 2074–2082, Barcelona, Spain, 2015.
[Yuan and Lin2006] Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society, 68(1):49–67, 2006.
[Zhang et al.2016] X. Zhang, J. Zou, K. He, and J. Sun. Accelerating very deep convolutional networks for classification and detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(10):1943–1955, 2016.
[Zhang et al.2017] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. 2017.

Structured Deep Neural Network Pruning by Varying Regularization Parameters

Authors

Structured Pruning for Efficient ConvNets via Incremental Regularization

Three Dimensional Convolutional Neural Network Pruning with Regularization-Based Method

Structured Probabilistic Pruning for Convolutional Neural Network Acceleration

High-dimensional Bayesian Optimization for CNN Auto Pruning with Clustering and Rollback

Recurrent Parameter Generators

Simultaneously Learning Architectures and Features of Deep Neural Networks

EDropout: Energy-Based Dropout and Pruning of Deep Neural Networks

1 Introduction

2 Related Work

3 The Proposed Method

3.1 Theoretical Analysis

Theorem 1

3.2 Method Description

3.3 Discussions

4 Experiments

4.1 Analysis with ConvNet on CIFAR-10

4.2 AlexNet on ImageNet

4.3 VGG-16 on ImageNet

4.4 ResNet-50 on ImageNet

5 Conclusion

Appendix A – Proof of Theorem 1

References

Structured Deep Neural Network Pruning by Varying Regularization Parameters

Authors

Structured Pruning for Efficient ConvNets via Incremental Regularization

Three Dimensional Convolutional Neural Network Pruning with Regularization-Based Method

Structured Probabilistic Pruning for Convolutional Neural Network Acceleration

High-dimensional Bayesian Optimization for CNN Auto Pruning with Clustering and Rollback

Recurrent Parameter Generators

Simultaneously Learning Architectures and Features of Deep Neural Networks

EDropout: Energy-Based Dropout and Pruning of Deep Neural Networks

This week in AI

1 Introduction

2 Related Work

3 The Proposed Method

3.1 Theoretical Analysis

Theorem 1

3.2 Method Description

3.3 Discussions

4 Experiments

4.1 Analysis with ConvNet on CIFAR-10

4.2 AlexNet on ImageNet

4.3 VGG-16 on ImageNet

4.4 ResNet-50 on ImageNet

5 Conclusion

Appendix A – Proof of Theorem 1

References