Log In Sign Up

Deep Mixtures of Factor Analysers

by   Yichuan Tang, et al.

An efficient way to learn deep density models that have many layers of latent variables is to learn one layer at a time using a model that has only one layer of latent variables. After learning each layer, samples from the posterior distributions for that layer are used as training data for learning the next layer. This approach is commonly used with Restricted Boltzmann Machines, which are undirected graphical models with a single hidden layer, but it can also be used with Mixtures of Factor Analysers (MFAs) which are directed graphical models. In this paper, we present a greedy layer-wise learning algorithm for Deep Mixtures of Factor Analysers (DMFAs). Even though a DMFA can be converted to an equivalent shallow MFA by multiplying together the factor loading matrices at different levels, learning and inference are much more efficient in a DMFA and the sharing of each lower-level factor loading matrix by many different higher level MFAs prevents overfitting. We demonstrate empirically that DMFAs learn better density models than both MFAs and two types of Restricted Boltzmann Machine on a wide variety of datasets.


Learning Restricted Boltzmann Machines via Influence Maximization

Graphical models are a rich language for describing high-dimensional dis...

From Boltzmann Machines to Neural Networks and Back Again

Graphical models are powerful tools for modeling high-dimensional data, ...

Training Restricted Boltzmann Machine by Perturbation

A new approach to maximum likelihood learning of discrete graphical mode...

Mixtures and products in two graphical models

We compare two statistical models of three binary random variables. One ...

Learning Restricted Boltzmann Machines with Arbitrary External Fields

We study the problem of learning graphical models with latent variables....

Is Joint Training Better for Deep Auto-Encoders?

Traditionally, when generative models of data are developed via deep arc...

Learning Restricted Boltzmann Machines with Few Latent Variables

Restricted Boltzmann Machines (RBMs) are a common family of undirected g...