Thanks for Nothing: Predicting Zero-Valued Activations with Lightweight Convolutional Neural Networks

09/17/2019
by   Gil Shomron, et al.
0

Convolutional neural networks (CNNs) introduce state-of-the-art results for various tasks with the price of high computational demands. Inspired by the observation that spatial correlation exists in CNN output feature maps (ofms), we propose a method to dynamically predict whether ofm activations are zero-valued or not according to their neighboring activation values, thereby avoiding zero-valued activations and reducing the number of convolution operations. We implement the zero activation predictor (ZAP) with a lightweight CNN, which imposes negligible overheads and is easy to deploy and train. Furthermore, the same ZAP can be tuned to many different operating points along the accuracy-savings trade-off curve. For example, using VGG-16 and the ILSVRC-2012 dataset, different operating points achieve a reduction of 23.5 and 32.3 degradation of 0.3 Considering one-epoch fine-tuning, 41.7 1.1

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 3

07/21/2018

Exploiting Spatial Correlation in Convolutional Neural Networks for Activation Value Prediction

Convolutional neural networks (CNNs) compute their output using weighted...
08/01/2019

Accelerating CNN Training by Sparsifying Activation Gradients

Gradients to activations get involved in most of the calculations during...
05/23/2017

SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks

Convolutional Neural Networks (CNNs) have emerged as a fundamental techn...
05/27/2019

SpecNet: Spectral Domain Convolutional Neural Network

The memory consumption of most Convolutional Neural Network (CNN) archit...
12/24/2018

Dynamic Runtime Feature Map Pruning

High bandwidth requirements are an obstacle for accelerating the trainin...
01/07/2020

Sparse Weight Activation Training

Training convolutional neural networks (CNNs) is time-consuming. Prior w...
10/17/2021

Compression-aware Projection with Greedy Dimension Reduction for Convolutional Neural Network Activations

Convolutional neural networks (CNNs) achieve remarkable performance in a...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.