A Survey on Deep Neural Network Compression: Challenges, Overview, and Solutions

10/05/2020
by   Rahul Mishra, et al.
0

Deep Neural Network (DNN) has gained unprecedented performance due to its automated feature extraction capability. This high order performance leads to significant incorporation of DNN models in different Internet of Things (IoT) applications in the past decade. However, the colossal requirement of computation, energy, and storage of DNN models make their deployment prohibitive on resource constraint IoT devices. Therefore, several compression techniques were proposed in recent years for reducing the storage and computation requirements of the DNN model. These techniques on DNN compression have utilized a different perspective for compressing DNN with minimal accuracy compromise. It encourages us to make a comprehensive overview of the DNN compression techniques. In this paper, we present a comprehensive review of existing literature on compressing DNN model that reduces both storage and computation requirements. We divide the existing approaches into five broad categories, i.e., network pruning, sparse representation, bits precision, knowledge distillation, and miscellaneous, based upon the mechanism incorporated for compressing the DNN model. The paper also discussed the challenges associated with each category of DNN compression techniques. Finally, we provide a quick summary of existing work under each category with the future direction in DNN compression.

READ FULL TEXT

page 1

page 3

page 14

research
09/30/2022

Designing and Training of Lightweight Neural Networks on Edge Devices using Early Halting in Knowledge Distillation

Automated feature extraction capability and significant performance of D...
research
03/14/2018

DeepN-JPEG: A Deep Neural Network Favorable JPEG-based Image Compression Framework

As one of most fascinating machine learning techniques, deep neural netw...
research
07/11/2022

Sparsifying Binary Networks

Binary neural networks (BNNs) have demonstrated their ability to solve c...
research
06/05/2020

An Overview of Neural Network Compression

Overparameterized networks trained to convergence have shown impressive ...
research
03/08/2020

A General Approach for Using Deep Neural Network for Digital Watermarking

Technologies of the Internet of Things (IoT) facilitate digital contents...
research
11/19/2021

Toward Compact Parameter Representations for Architecture-Agnostic Neural Network Compression

This paper investigates deep neural network (DNN) compression from the p...
research
08/13/2021

Pruning vs XNOR-Net: A Comprehensive Study of Deep Learning for Audio Classification on Edge-devices

Deep Learning has celebrated resounding successes in many application ar...

Please sign up or login with your details

Forgot password? Click here to reset