A Neighbourhood-Based Stopping Criterion for Contrastive Divergence Learning

07/24/2015
by   E. Romero, et al.
0

Restricted Boltzmann Machines (RBMs) are general unsupervised learning devices to ascertain generative models of data distributions. RBMs are often trained using the Contrastive Divergence learning algorithm (CD), an approximation to the gradient of the data log-likelihood. A simple reconstruction error is often used as a stopping criterion for CD, although several authors schulz-et-al-Convergence-Contrastive-Divergence-2010-NIPSw, fischer-igel-Divergence-Contrastive-Divergence-2010-ICANN have raised doubts concerning the feasibility of this procedure. In many cases the evolution curve of the reconstruction error is monotonic while the log-likelihood is not, thus indicating that the former is not a good estimator of the optimal stopping point for learning. However, not many alternatives to the reconstruction error have been discussed in the literature. In this manuscript we investigate simple alternatives to the reconstruction error, based on the inclusion of information contained in neighboring states to the training set, as a stopping criterion for CD learning.

READ FULL TEXT
research
11/22/2017

From Monte Carlo to Las Vegas: Improving Restricted Boltzmann Machine Training Through Stopping Sets

We propose a Las Vegas transformation of Markov Chain Monte Carlo (MCMC)...
research
05/07/2020

Training and Classification using a Restricted Boltzmann Machine on the D-Wave 2000Q

Restricted Boltzmann Machine (RBM) is an energy based, undirected graphi...
research
09/21/2017

Learning RBM with a DC programming Approach

By exploiting the property that the RBM log-likelihood function is the d...
research
10/06/2015

Population-Contrastive-Divergence: Does Consistency help with RBM training?

Estimating the log-likelihood gradient with respect to the parameters of...
research
05/19/2016

Recurrent Exponential-Family Harmoniums without Backprop-Through-Time

Exponential-family harmoniums (EFHs), which extend restricted Boltzmann ...
research
02/24/2022

Clarifying MCMC-based training of modern EBMs : Contrastive Divergence versus Maximum Likelihood

The Energy-Based Model (EBM) framework is a very general approach to gen...
research
02/11/2021

Learning Gaussian-Bernoulli RBMs using Difference of Convex Functions Optimization

The Gaussian-Bernoulli restricted Boltzmann machine (GB-RBM) is a useful...

Please sign up or login with your details

Forgot password? Click here to reset