BinaryConnect: Training Deep Neural Networks with binary weights during propagations

11/02/2015
by   Matthieu Courbariaux, et al.
0

Deep Neural Networks (DNN) have achieved state-of-the-art results in a wide range of tasks, with the best results obtained with large training sets and large models. In the past, GPUs enabled these breakthroughs because of their greater computational speed. In the future, faster computation at both training and test time is likely to be crucial for further progress and for consumer applications on low-power devices. As a result, there is much interest in research and development of dedicated hardware for Deep Learning (DL). Binary weights, i.e., weights which are constrained to only two possible values (e.g. -1 or 1), would bring great benefits to specialized DL hardware by replacing many multiply-accumulate operations by simple accumulations, as multipliers are the most space and power-hungry components of the digital implementation of neural networks. We introduce BinaryConnect, a method which consists in training a DNN with binary weights during the forward and backward propagations, while retaining precision of the stored weights in which gradients are accumulated. Like other dropout schemes, we show that BinaryConnect acts as regularizer and we obtain near state-of-the-art results with BinaryConnect on the permutation-invariant MNIST, CIFAR-10 and SVHN.

READ FULL TEXT
research
03/22/2022

FxP-QNet: A Post-Training Quantizer for the Design of Mixed Low-Precision DNNs with Dynamic Fixed-Point Representation

Deep neural networks (DNNs) have demonstrated their effectiveness in a w...
research
10/21/2017

Learning Discrete Weights Using the Local Reparameterization Trick

Recent breakthroughs in computer vision make use of large deep neural ne...
research
12/31/2018

BNN+: Improved Binary Network Training

Deep neural networks (DNN) are widely used in many applications. However...
research
02/08/2016

Binarized Neural Networks

We introduce a method to train Binarized Neural Networks (BNNs) - neural...
research
10/07/2013

Mean Field Bayes Backpropagation: scalable training of multilayer neural networks with binary weights

Significant success has been reported recently using deep neural network...
research
11/29/2017

Transfer Learning with Binary Neural Networks

Previous work has shown that it is possible to train deep neural network...
research
02/21/2022

Enabling On-Device Smartphone GPU based Training: Lessons Learned

Deep Learning (DL) has shown impressive performance in many mobile appli...

Please sign up or login with your details

Forgot password? Click here to reset