Local Maxima in the Likelihood of Gaussian Mixture Models: Structural Results and Algorithmic Consequences

09/04/2016
by   Chi Jin, et al.
0

We provide two fundamental results on the population (infinite-sample) likelihood function of Gaussian mixture models with M ≥ 3 components. Our first main result shows that the population likelihood function has bad local maxima even in the special case of equally-weighted mixtures of well-separated and spherical Gaussians. We prove that the log-likelihood value of these bad local maxima can be arbitrarily worse than that of any global optimum, thereby resolving an open question of Srebro (2007). Our second main result shows that the EM algorithm (or a first-order variant of it) with random initialization will converge to bad critical points with probability at least 1-e^-Ω(M). We further establish that a first-order variant of EM will not converge to strict saddle points almost surely, indicating that the poor performance of the first-order method can be attributed to the existence of bad local maxima rather than bad saddle points. Overall, our results highlight the necessity of careful initialization when using the EM algorithm in practice, even when applied in highly favorable settings.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/08/2019

Comparing EM with GD in Mixture Models of Two Components

The expectation-maximization (EM) algorithm has been widely used in mini...
research
09/28/2020

Likelihood Landscape and Local Minima Structures of Gaussian Mixture Models

In this paper, we study the landscape of the population negative log-lik...
research
03/23/2012

k-MLE: A fast algorithm for learning statistical mixture models

We describe k-MLE, a fast and efficient local search algorithm for learn...
research
05/23/2022

Beyond EM Algorithm on Over-specified Two-Component Location-Scale Gaussian Mixtures

The Expectation-Maximization (EM) algorithm has been predominantly used ...
research
10/28/2018

Iteratively Learning from the Best

We study a simple generic framework to address the issue of bad training...
research
07/25/2020

Fair Marriage Principle and Initialization Map for the EM Algorithm

The popular convergence theory of the EM algorithm explains that the obs...
research
02/10/2023

Efficient and Accurate Learning of Mixtures of Plackett-Luce Models

Mixture models of Plackett-Luce (PL) – one of the most fundamental ranki...

Please sign up or login with your details

Forgot password? Click here to reset