Nonconvex Regularization for Network Slimming:Compressing CNNs Even More
share
In the last decade, convolutional neural networks (CNNs) have evolved to become the dominant models for various computer vision tasks, but they cannot be deployed in low-memory devices due to its high memory requirement and computational cost. One popular, straightforward approach to compressing CNNs is network slimming, which imposes an ℓ_1 penalty on the channel-associated scaling factors in the batch normalization layers during training. In this way, channels with low scaling factors are identified to be insignificant and are pruned in the models. In this paper, we propose replacing the ℓ_1 penalty with the ℓ_p and transformed ℓ_1 (Tℓ_1) penalties since these nonconvex penalties outperformed ℓ_1 in yielding sparser satisfactory solutions in various compressed sensing problems. In our numerical experiments, we demonstrate network slimming with ℓ_p and Tℓ_1 penalties on VGGNet and Densenet trained on CIFAR 10/100. The results demonstrate that the nonconvex penalties compress CNNs better than ℓ_1. In addition, Tℓ_1 preserves the model accuracy after channel pruning, and ℓ_1/2, 3/4 yield compressed models with similar accuracies as ℓ_1 after retraining.
READ FULL TEXT