Training Binary Neural Networks using the Bayesian Learning Rule

by   Xiangming Meng, et al.

Neural networks with binary weights are computation-efficient and hardware-friendly, but their training is challenging because it involves a discrete optimization problem. Surprisingly, ignoring the discrete nature of the problem and using gradient-based methods, such as Straight-Through Estimator, still works well in practice. This raises the question: are there principled approaches which justify such methods? In this paper, we propose such an approach using the Bayesian learning rule. The rule, when applied to estimate a Bernoulli distribution over the binary weights, results in an algorithm which justifies some of the algorithmic choices made by the previous approaches. The algorithm not only obtains state-of-the-art performance, but also enables uncertainty estimation for continual learning to avoid catastrophic forgetting. Our work provides a principled approach for training binary neural networks which justifies and extends existing approaches.


Uncertainty-guided Continual Learning with Bayesian Neural Networks

Continual learning aims to learn new tasks without forgetting previously...

On Sequential Bayesian Inference for Continual Learning

Sequential Bayesian inference can be used for continual learning to prev...

AdaSTE: An Adaptive Straight-Through Estimator to Train Binary Neural Networks

We propose a new algorithm for training deep neural networks (DNNs) with...

Continual learning: a feature extraction formalization, an efficient algorithm, and fundamental obstructions

Continual learning is an emerging paradigm in machine learning, wherein ...

Reintroducing Straight-Through Estimators as Principled Methods for Stochastic Binary Networks

Training neural networks with binary weights and activations is a challe...

Learning Discrete Weights Using the Local Reparameterization Trick

Recent breakthroughs in computer vision make use of large deep neural ne...

The Automatic Training of Rule Bases that Use Numerical Uncertainty Representations

The use of numerical uncertainty representations allows better modeling ...

Code Repositories


Code for the paper "Training Binary Neural Networks with Bayesian Learning Rule

view repo

Please sign up or login with your details

Forgot password? Click here to reset