Coordinating Filters for Faster Deep Neural Networks

03/28/2017
by   Wei Wen, et al.
0

Very large-scale Deep Neural Networks (DNNs) have achieved remarkable successes in a large variety of computer vision tasks. However, the high computation intensity of DNNs makes it challenging to deploy these models on resource-limited systems. Some studies used low-rank approaches that approximate the filters by low-rank basis to accelerate the testing. Those works directly decomposed the pre-trained DNNs by Low-Rank Approximations (LRA). How to train DNNs toward lower-rank space for more efficient DNNs, however, remains as an open area. To solve the issue, in this work, we propose Force Regularization, which uses attractive forces to enforce filters so as to coordinate more weight information into lower-rank space. We mathematically and empirically verify that after applying our technique, standard LRA methods can reconstruct filters using much lower basis and thus result in faster DNNs. The effectiveness of our approach is comprehensively evaluated in ResNets, AlexNet, and GoogLeNet. In AlexNet, for example, Force Regularization gains 2x speedup on modern GPU without accuracy loss and 4.05x speedup on CPU by paying small accuracy degradation. Moreover, Force Regularization better initializes the low-rank DNNs such that the fine-tuning can converge faster toward higher accuracy. The obtained lower-rank DNNs can be further sparsified, proving that Force Regularization can be integrated with state-of-the-art sparsity-based acceleration methods. Source code is available in https://github.com/wenwei202/caffe

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/09/2019

Traned Rank Pruning for Efficient Deep Neural Networks

To accelerate DNNs inference, low-rank approximation has been widely ado...
research
12/06/2018

Trained Rank Pruning for Efficient Deep Neural Networks

The performance of Deep Neural Networks (DNNs) keeps elevating in recent...
research
05/18/2018

Stop memorizing: A data-dependent regularization framework for intrinsic pattern learning

Deep neural networks (DNNs) typically have enough capacity to fit random...
research
08/12/2016

Learning Structured Sparsity in Deep Neural Networks

High demand for computation resources severely hinders deployment of lar...
research
05/15/2014

Speeding up Convolutional Neural Networks with Low Rank Expansions

The focus of this paper is speeding up the evaluation of convolutional n...
research
07/26/2018

A Unified Approximation Framework for Non-Linear Deep Neural Networks

Deep neural networks (DNNs) have achieved significant success in a varie...
research
08/28/2023

Maestro: Uncovering Low-Rank Structures via Trainable Decomposition

Deep Neural Networks (DNNs) have been a large driver and enabler for AI ...

Please sign up or login with your details

Forgot password? Click here to reset