Per-Tensor Fixed-Point Quantization of the Back-Propagation Algorithm

12/31/2018
by   Charbel Sakr, et al.
0

The high computational and parameter complexity of neural networks makes their training very slow and difficult to deploy on energy and storage-constrained computing systems. Many network complexity reduction techniques have been proposed including fixed-point implementation. However, a systematic approach for designing full fixed-point training and inference of deep neural networks remains elusive. We describe a precision assignment methodology for neural network training in which all network parameters, i.e., activations and weights in the feedforward path, gradients and weight accumulators in the feedback path, are assigned close to minimal precision. The precision assignment is derived analytically and enables tracking the convergence behavior of the full precision training, known to converge a priori. Thus, our work leads to a systematic methodology of determining suitable precision for fixed-point training. The near optimality (minimality) of the resulting precision assignment is validated empirically for four networks on the CIFAR-10, CIFAR-100, and SVHN datasets. The complexity reduction arising from our approach is compared with other fixed-point neural network designs.

READ FULL TEXT
research
02/09/2015

Deep Learning with Limited Numerical Precision

Training of large-scale deep neural networks is often constrained by the...
research
12/22/2014

Training deep neural networks with low precision multiplications

Multipliers are the most space and power-hungry arithmetic operators of ...
research
02/13/2018

Training and Inference with Integers in Deep Neural Networks

Researches on deep neural networks with discrete parameters and their de...
research
06/01/2017

Dynamic Stripes: Exploiting the Dynamic Precision Requirements of Activation Values in Neural Networks

Stripes is a Deep Neural Network (DNN) accelerator that uses bit-serial ...
research
09/06/2020

A systematic approach to computing and indexing the fixed points of an iterated exponential

This paper describes a systematic method of numerically computing and in...
research
11/07/2016

Fixed-point Factorized Networks

In recent years, Deep Neural Networks (DNN) based methods have achieved ...
research
05/25/2023

Banana: Banach Fixed-Point Network for Pointcloud Segmentation with Inter-Part Equivariance

Equivariance has gained strong interest as a desirable network property ...

Please sign up or login with your details

Forgot password? Click here to reset