Divergence-Based Motivation for Online EM and Combining Hidden Variable Models

02/11/2019
by   Ehsan Amid, et al.
0

Expectation-Maximization (EM) is the fallback method for parameter estimation of hidden (aka latent) variable models. Given the full batch of data, EM forms an upper-bound of the negative log-likelihood of the model at each iteration and then updates to the minimizer of this upper-bound. We introduce a versatile online variant of EM where the data arrives in as a stream. Our motivation is based on the relative entropy divergences between two joint distributions over the hidden and visible variables. We view the EM upper-bound as a Monte Carlo approximation of an expectation and show that the joint relative entropy divergence induces a similar expectation form. As a result, we employ the divergence to the old model as the inertia term to motivate our online EM algorithm. Our motivation is more widely applicable than previous ones and leads to simple online updates for mixture of exponential distributions, hidden Markov models, and the first known online update for Kalman filters. Additionally, the finite sample form of the inertia term lets us derive online updates when there is no closed form solution. Experimentally, sweeping the data with an online update converges much faster than the batch update. Our divergence based methods also lead to a simple way to combine hidden variable models and this immediately gives efficient algorithms for distributed setting.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/09/2018

Efficient convergence through adaptive learning in sequential Monte Carlo Expectation Maximization

Expectation maximization (EM) is a technique for estimating maximum-like...
research
01/11/2014

An Online Expectation-Maximisation Algorithm for Nonnegative Matrix Factorisation Models

In this paper we formulate the nonnegative matrix factorisation (NMF) pr...
research
11/21/2022

EM's Convergence in Gaussian Latent Tree Models

We study the optimization landscape of the log-likelihood function and t...
research
05/28/2020

Joint Stochastic Approximation and Its Application to Learning Discrete Latent Variable Models

Although with progress in introducing auxiliary amortized inference mode...
research
09/21/2017

SpectralFPL: Online Spectral Learning for Single Topic Models

This paper studies how to efficiently learn an optimal latent variable m...
research
11/02/2020

Homeomorphic-Invariance of EM: Non-Asymptotic Convergence in KL Divergence for Exponential Families via Mirror Descent

Expectation maximization (EM) is the default algorithm for fitting proba...
research
10/23/2020

Overcoming Conflicting Data for Model Updates

In this paper, we explore how to use a small amount of new data to updat...

Please sign up or login with your details

Forgot password? Click here to reset