Mode-Assisted Unsupervised Learning of Restricted Boltzmann Machines

01/15/2020
by   Haik Manukian, et al.
University of California, San Diego
15

Restricted Boltzmann machines (RBMs) are a powerful class of generative models, but their training requires computing a gradient that, unlike supervised backpropagation on typical loss functions, is notoriously difficult even to approximate. Here, we show that properly combining standard gradient updates with an off-gradient direction, constructed from samples of the RBM ground state (mode), improves their training dramatically over traditional gradient methods. This approach, which we call mode training, promotes faster training and stability, in addition to lower converged relative entropy (KL divergence). Along with the proofs of stability and convergence of this method, we also demonstrate its efficacy on synthetic datasets where we can compute KL divergences exactly, as well as on a larger machine learning standard, MNIST. The mode training we suggest is quite versatile, as it can be applied in conjunction with any given gradient method, and is easily extended to more general energy-based neural network structures such as deep, convolutional and unrestricted Boltzmann machines.

READ FULL TEXT

page 1

page 2

page 3

page 4

02/17/2021

Mode-Assisted Joint Training of Deep Boltzmann Machines

The deep extension of the restricted Boltzmann machine (RBM), known as t...
11/27/2020

Tractable loss function and color image generation of multinary restricted Boltzmann machine

The restricted Boltzmann machine (RBM) is a representative generative mo...
07/07/2015

Wasserstein Training of Boltzmann Machines

The Boltzmann machine provides a useful framework to learn highly comple...
09/02/2022

Three Learning Stages and Accuracy-Efficiency Tradeoff of Restricted Boltzmann Machines

Restricted Boltzmann Machines (RBMs) offer a versatile architecture for ...
12/09/2022

Attention in a family of Boltzmann machines emerging from modern Hopfield networks

Hopfield networks and Boltzmann machines (BMs) are fundamental energy-ba...

Please sign up or login with your details

Forgot password? Click here to reset