CPT: Efficient Deep Neural Network Training via Cyclic Precision

01/25/2021
by   Yonggan Fu, et al.
0

Low-precision deep neural network (DNN) training has gained tremendous attention as reducing precision is one of the most effective knobs for boosting DNNs' training time/energy efficiency. In this paper, we attempt to explore low-precision training from a new perspective as inspired by recent findings in understanding DNN training: we conjecture that DNNs' precision might have a similar effect as the learning rate during DNN training, and advocate dynamic precision along the training trajectory for further boosting the time/energy efficiency of DNN training. Specifically, we propose Cyclic Precision Training (CPT) to cyclically vary the precision between two boundary values which can be identified using a simple precision range test within the first few training epochs. Extensive simulations and ablation studies on five datasets and ten models demonstrate that CPT's effectiveness is consistent across various models/tasks (including classification and language modeling). Furthermore, through experiments and visualization we show that CPT helps to (1) converge to a wider minima with a lower generalization error and (2) reduce training variance which we believe opens up a new design knob for simultaneously improving the optimization and efficiency of DNN training. Our codes are available at: https://github.com/RICE-EIC/CPT.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/24/2020

FracTrain: Fractionally Squeezing Bit Savings Both Temporally and Spatially for Efficient DNN Training

Recent breakthroughs in deep neural networks (DNNs) have fueled a tremen...
research
03/15/2022

LDP: Learnable Dynamic Precision for Efficient Deep Neural Network Training and Inference

Low precision deep neural network (DNN) training is one of the most effe...
research
09/06/2019

Training Deep Neural Networks Using Posit Number System

With the increasing size of Deep Neural Network (DNN) models, the high m...
research
06/02/2022

DepthShrinker: A New Compression Paradigm Towards Boosting Real-Hardware Efficiency of Compact Neural Networks

Efficient deep neural network (DNN) models equipped with compact operato...
research
09/05/2019

Detecting Deep Neural Network Defects with Data Flow Analysis

Deep neural networks (DNNs) are shown to be promising solutions in many ...
research
07/11/2022

Deep neural network based adaptive learning for switched systems

In this paper, we present a deep neural network based adaptive learning ...
research
06/02/2022

Fast Benchmarking of Accuracy vs. Training Time with Cyclic Learning Rates

Benchmarking the tradeoff between neural network accuracy and training t...

Please sign up or login with your details

Forgot password? Click here to reset