Adiabatic Persistent Contrastive Divergence Learning

05/26/2016
by   Hyeryung Jang, et al.
0

This paper studies the problem of parameter learning in probabilistic graphical models having latent variables, where the standard approach is the expectation maximization algorithm alternating expectation (E) and maximization (M) steps. However, both E and M steps are computationally intractable for high dimensional data, while the substitution of one step to a faster surrogate for combating against intractability can often cause failure in convergence. We propose a new learning algorithm which is computationally efficient and provably ensures convergence to a correct optimum. Its key idea is to run only a few cycles of Markov Chains (MC) in both E and M steps. Such an idea of running incomplete MC has been well studied only for M step in the literature, called Contrastive Divergence (CD) learning. While such known CD-based schemes find approximated gradients of the log-likelihood via the mean-field approach in E step, our proposed algorithm does exact ones via MC algorithms in both steps due to the multi-time-scale stochastic approximation theory. Despite its theoretical guarantee in convergence, the proposed scheme might suffer from the slow mixing of MC in E step. To tackle it, we also propose a hybrid approach applying both mean-field and MC approximation in E step, where the hybrid approach outperforms the bare mean-field CD scheme in our experiments on real-world datasets.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/17/2020

A Mean Field Games model for finite mixtures of Bernoulli distributions

Finite mixture models are an important tool in the statistical analysis ...
research
05/28/2020

Joint Stochastic Approximation and Its Application to Learning Discrete Latent Variable Models

Although with progress in introducing auxiliary amortized inference mode...
research
06/01/2011

Mean Field Methods for a Special Class of Belief Networks

The chief aim of this paper is to propose mean-field approximations for ...
research
07/04/2019

A Mean Field Games approach to Cluster Analysis

In this paper, we develop a Mean Field Games approach to Cluster Analysi...
research
10/19/2020

Robust High Dimensional Expectation Maximization Algorithm via Trimmed Hard Thresholding

In this paper, we study the problem of estimating latent variable models...
research
12/19/2016

An extended Perona-Malik model based on probabilistic models

The Perona-Malik model has been very successful at restoring images from...
research
03/23/2020

Deterministic Approximate EM Algorithm; Application to the Riemann Approximation EM and the Tempered EM

The Expectation Maximisation (EM) algorithm is widely used to optimise n...

Please sign up or login with your details

Forgot password? Click here to reset