Approximate Random Dropout

05/23/2018
by   Zhuoran Song, et al.
0

The training phases of Deep neural network (DNN) consume enormous processing time and energy. Compression techniques for inference acceleration leveraging the sparsity of DNNs, however, can be hardly used in the training phase. Because the training involves dense matrix-multiplication using GPGPU, which endorse regular and structural data layout. In this paper, we exploit the sparsity of DNN resulting from the random dropout technique to eliminate the unnecessary computation and data access for those dropped neurons or synapses in the training phase. Experiments results on MLP and LSTM on standard benchmarks show that the proposed Approximate Random Dropout can reduce the training time by half on average with ignorable accuracy loss.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/11/2022

DNN Training Acceleration via Exploring GPGPU Friendly Sparsity

The training phases of Deep neural network (DNN) consumes enormous proce...
research
05/23/2018

Excitation Dropout: Encouraging Plasticity in Deep Neural Networks

We propose a guided dropout regularizer for deep networks based on the e...
research
08/29/2018

Dropout with Tabu Strategy for Regularizing Deep Neural Networks

Dropout has proven to be an effective technique for regularization and p...
research
07/21/2020

SliceOut: Training Transformers and CNNs faster while using less memory

We demonstrate 10-40 EfficientNets, and Transformer models, with minimal...
research
11/03/2017

SparseNN: An Energy-Efficient Neural Network Accelerator Exploiting Input and Output Sparsity

Contemporary Deep Neural Network (DNN) contains millions of synaptic con...
research
11/28/2020

Optical Phase Dropout in Diffractive Deep Neural Network

Unitary learning is a backpropagation that serves to unitary weights upd...
research
10/10/2018

LIRS: Enabling efficient machine learning on NVM-based storage via a lightweight implementation of random shuffling

Machine learning algorithms, such as Support Vector Machine (SVM) and De...

Please sign up or login with your details

Forgot password? Click here to reset