Training Restricted Boltzmann Machines with Binary Synapses using the Bayesian Learning Rule

07/09/2020
by   Xiangming Meng, et al.
The University of Tokyo
9

Restricted Boltzmann machines (RBMs) with low-precision synapses are much appealing with high energy efficiency. However, training RBMs with binary synapses is challenging due to the discrete nature of synapses. Recently Huang proposed one efficient method to train RBMs with binary synapses by using a combination of gradient ascent and the message passing algorithm under the variational inference framework. However, additional heuristic clipping operation is needed. In this technical note, inspired from Huang's work , we propose one alternative optimization method using the Bayesian learning rule, which is one natural gradient variational inference method. As opposed to Huang's method, we update the natural parameters of the variational symmetric Bernoulli distribution rather than the expectation parameters. Since the natural parameters take values in the entire real domain, no additional clipping is needed. Interestingly, the algorithm in <cit.> could be viewed as one first-order approximation of the proposed algorithm, which justifies its efficacy with heuristic clipping.

READ FULL TEXT

page 1

page 2

page 3

page 4

03/15/2018

Variational Message Passing with Structured Inference Networks

Recent efforts on combining deep models with probabilistic graphical mod...
03/31/2022

Quantum-Aided Meta-Learning for Bayesian Binary Neural Networks via Born Machines

Near-term noisy intermediate-scale quantum circuits can efficiently impl...
05/15/2020

Minimax formula for the replica symmetric free energy of deep restricted Boltzmann machines

We study the free energy of a most used deep architecture for restricted...
02/23/2015

Approximate Message Passing with Restricted Boltzmann Machine Priors

Approximate Message Passing (AMP) has been shown to be an excellent stat...
01/23/2014

Gaussian-binary Restricted Boltzmann Machines on Modeling Natural Image Statistics

We present a theoretical analysis of Gaussian-binary restricted Boltzman...
01/16/2013

Metric-Free Natural Gradient for Joint-Training of Boltzmann Machines

This paper introduces the Metric-Free Natural Gradient (MFNG) algorithm ...

Please sign up or login with your details

Forgot password? Click here to reset