Learning Restricted Boltzmann Machines with Few Latent Variables

by   Guy Bresler, et al.

Restricted Boltzmann Machines (RBMs) are a common family of undirected graphical models with latent variables. An RBM is described by a bipartite graph, with all observed variables in one layer and all latent variables in the other. We consider the task of learning an RBM given samples generated according to it. The best algorithms for this task currently have time complexity Õ(n^2) for ferromagnetic RBMs (i.e., with attractive potentials) but Õ(n^d) for general RBMs, where n is the number of observed variables and d is the maximum degree of a latent variable. Let the MRF neighborhood of an observed variable be its neighborhood in the Markov Random Field of the marginal distribution of the observed variables. In this paper, we give an algorithm for learning general RBMs with time complexity Õ(n^2^s+1), where s is the maximum number of latent variables connected to the MRF neighborhood of an observed variable. This represents an improvement when s < log_2 (d-1), which is satisfied by many classes of RBMs with "few latent variables”. Furthermore, we give a version of this learning algorithm that recovers a model with small prediction error and whose sample complexity is independent of the minimum potential in the Markov Random Field of the observed variables. This is of interest because the sample complexity of current algorithms scales with the inverse of the minimum potential, which cannot be controlled in terms of natural properties of the RBM.


page 1

page 2

page 3

page 4


Learning Restricted Boltzmann Machines via Influence Maximization

Graphical models are a rich language for describing high-dimensional dis...

Learning Restricted Boltzmann Machines with Arbitrary External Fields

We study the problem of learning graphical models with latent variables....

Learning loopy graphical models with latent variables: Efficient methods and guarantees

The problem of structure estimation in graphical models with latent vari...

A Latent-Variable Lattice Model

Markov random field (MRF) learning is intractable, and its approximation...

Causal Channels

We consider causal models with two observed variables and one latent var...

Hardness of Identity Testing for Restricted Boltzmann Machines and Potts models

We study identity testing for restricted Boltzmann machines (RBMs), and ...

Towards sharp error analysis of extended Lagrangian molecular dynamics

The extended Lagrangian molecular dynamics (XLMD) method provides a usef...