DeepAI
Log In Sign Up

Homeomorphic-Invariance of EM: Non-Asymptotic Convergence in KL Divergence for Exponential Families via Mirror Descent

11/02/2020
by   Frederik Kunstner, et al.
0

Expectation maximization (EM) is the default algorithm for fitting probabilistic models with missing or latent variables, yet we lack a full understanding of its non-asymptotic convergence properties. Previous works show results along the lines of "EM converges at least as fast as gradient descent" by assuming the conditions for the convergence of gradient descent apply to EM. This approach is not only loose, in that it does not capture that EM can make more progress than a gradient step, but the assumptions fail to hold for textbook examples of EM like Gaussian mixtures. In this work we first show that for the common setting of exponential family distributions, viewing EM as a mirror descent algorithm leads to convergence rates in Kullback-Leibler (KL) divergence. Then, we show how the KL divergence is related to first-order stationarity via Bregman divergences. In contrast to previous works, the analysis is invariant to the choice of parametrization and holds with minimal assumptions. We also show applications of these ideas to local linear (and superlinear) convergence rates, generalized EM, and non-exponential family distributions.

READ FULL TEXT

page 1

page 2

page 3

page 4

06/17/2022

Mirror Descent with Relative Smoothness in Measure Spaces, with application to Sinkhorn and EM

Many problems in machine learning can be formulated as optimizing a conv...
05/17/2022

On the Convergence of Policy in Unregularized Policy Mirror Descent

In this short note, we give the convergence analysis of the policy in th...
10/12/2018

Global Convergence of EM Algorithm for Mixtures of Two Component Linear Regression

The Expectation-Maximization algorithm is perhaps the most broadly used ...
01/07/2022

Bregman divergence based em algorithm and its application to classical and quantum rate distortion theory

We formulate em algorithm in the framework of Bregman divergence, which ...
06/07/2020

Learning Behaviors with Uncertain Human Feedback

Human feedback is widely used to train agents in many domains. However, ...
09/07/2022

Riemannian optimization for non-centered mixture of scaled Gaussian distributions

This paper studies the statistical model of the non-centered mixture of ...
11/12/2021

Convergence Rates for the MAP of an Exponential Family and Stochastic Mirror Descent – an Open Problem

We consider the problem of upper bounding the expected log-likelihood su...