Bit-wise Training of Neural Network Weights

02/19/2022
by   Cristian Ivan, et al.
0

We introduce an algorithm where the individual bits representing the weights of a neural network are learned. This method allows training weights with integer values on arbitrary bit-depths and naturally uncovers sparse networks, without additional constraints or regularization techniques. We show better results than the standard training technique with fully connected networks and similar performance as compared to standard training for convolutional and residual networks. By training bits in a selective manner we found that the biggest contribution to achieving high accuracy is given by the first three most significant bits, while the rest provide an intrinsic regularization. As a consequence more than 90% of a network can be used to store arbitrary codes without affecting its accuracy. These codes may be random noise, binary files or even the weights of previously trained networks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/09/2020

FleXOR: Trainable Fractional Quantization

Quantization based on the binary codes is gaining attention because each...
research
01/26/2022

Post-training Quantization for Neural Networks with Provable Guarantees

While neural networks have been remarkably successful in a wide array of...
research
02/03/2020

Exponential discretization of weights of neural network connections in pre-trained neural networks

To reduce random access memory (RAM) requirements and to increase speed ...
research
06/20/2018

Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?

Binary neural networks (BNN) have been studied extensively since they ru...
research
12/07/2021

BDFA: A Blind Data Adversarial Bit-flip Attack on Deep Neural Networks

Adversarial bit-flip attack (BFA) on Neural Network weights can result i...
research
11/30/2019

A binary-activation, multi-level weight RNN and training algorithm for processing-in-memory inference with eNVM

We present a new algorithm for training neural networks with binary acti...
research
05/20/2015

A Max-Sum algorithm for training discrete neural networks

We present an efficient learning algorithm for the problem of training n...

Please sign up or login with your details

Forgot password? Click here to reset