On the Global Convergence of (Fast) Incremental Expectation Maximization Methods

by   Belhal Karimi, et al.

The EM algorithm is one of the most popular algorithm for inference in latent data models. The original formulation of the EM algorithm does not scale to large data set, because the whole data set is required at each iteration of the algorithm. To alleviate this problem, Neal and Hinton have proposed an incremental version of the EM (iEM) in which at each iteration the conditional expectation of the latent data (E-step) is updated only for a mini-batch of observations. Another approach has been proposed by Cappé and Moulines in which the E-step is replaced by a stochastic approximation step, closely related to stochastic gradient. In this paper, we analyze incremental and stochastic version of the EM algorithm as well as the variance reduced-version of Chen et. al. in a common unifying framework. We also introduce a new version incremental version, inspired by the SAGA algorithm by Defazio et. al. We establish non-asymptotic convergence bounds for global convergence. Numerical applications are presented in this article to illustrate our findings.


page 1

page 2

page 3

page 4


The Perturbed Prox-Preconditioned SPIDER algorithm for EM-based large scale learning

Incremental Expectation Maximization (EM) algorithms were introduced to ...

Fast Incremental Expectation Maximization for finite-sum optimization: nonasymptotic convergence

Fast Incremental Expectation Maximization (FIEM) is a version of the EM ...

Properties of the Stochastic Approximation EM Algorithm with Mini-batch Sampling

To speed up convergence a mini-batch version of the Monte Carlo Markov C...

Stochastic EM methods with Variance Reduction for Penalised PET Reconstructions

Expectation-maximization (EM) is a popular and well-established method f...

An Incremental Slicing Method for Functional Programs

Several applications of slicing require a program to be sliced with resp...

Expectation-Propogation for the Generative Aspect Model

The generative aspect model is an extension of the multinomial model for...

EM algorithms for ICA

Independent component analysis (ICA) is a widely spread data exploration...