Convergence Rates for the MAP of an Exponential Family and Stochastic Mirror Descent – an Open Problem

by   Rémi Le Priol, et al.

We consider the problem of upper bounding the expected log-likelihood sub-optimality of the maximum likelihood estimate (MLE), or a conjugate maximum a posteriori (MAP) for an exponential family, in a non-asymptotic way. Surprisingly, we found no general solution to this problem in the literature. In particular, current theories do not hold for a Gaussian or in the interesting few samples regime. After exhibiting various facets of the problem, we show we can interpret the MAP as running stochastic mirror descent (SMD) on the log-likelihood. However, modern convergence results do not apply for standard examples of the exponential family, highlighting holes in the convergence literature. We believe solving this very fundamental problem may bring progress to both the statistics and optimization communities.



There are no comments yet.


page 18

page 19

page 20


Maximum likelihood estimation for matrix normal models via quiver representations

In this paper, we study the log-likelihood function and Maximum Likeliho...

A Polynomial Time Algorithm for Log-Concave Maximum Likelihood via Locally Exponential Families

We consider the problem of computing the maximum likelihood multivariate...

Marginal Weighted Maximum Log-likelihood for Efficient Learning of Perturb-and-Map models

We consider the structured-output prediction problem through probabilist...

Database Alignment with Gaussian Features

We consider the problem of aligning a pair of databases with jointly Gau...

Homeomorphic-Invariance of EM: Non-Asymptotic Convergence in KL Divergence for Exponential Families via Mirror Descent

Expectation maximization (EM) is the default algorithm for fitting proba...

Adaptive exponential power distribution with moving estimator for nonstationary time series

While standard estimation assumes that all datapoints are from probabili...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.