Expected Information Maximization: Using the I-Projection for Mixture Density Estimation

by   Philipp Becker, et al.

Modelling highly multi-modal data is a challenging problem in machine learning. Most algorithms are based on maximizing the likelihood, which corresponds to the M(oment)-projection of the data distribution to the model distribution. The M-projection forces the model to average over modes it cannot represent. In contrast, the I(information)-projection ignores such modes in the data and concentrates on the modes the model can represent. Such behavior is appealing whenever we deal with highly multi-modal data where modelling single modes correctly is more important than covering all the modes. Despite this advantage, the I-projection is rarely used in practice due to the lack of algorithms that can efficiently optimize it based on data. In this work, we present a new algorithm called Expected Information Maximization (EIM) for computing the I-projection solely based on samples for general latent variable models, where we focus on Gaussian mixtures models and Gaussian mixtures of experts. Our approach applies a variational upper bound to the I-projection objective which decomposes the original objective into single objectives for each mixture component as well as for the coefficients, allowing an efficient optimization. Similar to GANs, our approach employs discriminators but uses a more stable optimization procedure, using a tight upper bound. We show that our algorithm is much more effective in computing the I-projection than recent GAN approaches and we illustrate the effectiveness of our approach for modelling multi-modal behavior on two pedestrian and traffic prediction datasets.


Modal clustering on PPGMMGA projection subspace

PPGMMGA is a Projection Pursuit (PP) algorithm aimed at detecting and vi...

Latent Gaussian Process Regression

We introduce Latent Gaussian Process Regression which is a latent variab...

Projection pursuit based on Gaussian mixtures and evolutionary algorithms

We propose a projection pursuit (PP) algorithm based on Gaussian mixture...

A fast and efficient Modal EM algorithm for Gaussian mixtures

In the modal approach to clustering, clusters are defined as the local m...

Information Maximizing Curriculum: A Curriculum-Based Approach for Training Mixtures of Experts

Mixtures of Experts (MoE) are known for their ability to learn complex c...

Unsupervised K-modal Styled Content Generation

The emergence of generative models based on deep neural networks has rec...

How many modes can a constrained Gaussian mixture have?

We show, by an explicit construction, that a mixture of univariate Gauss...

Please sign up or login with your details

Forgot password? Click here to reset