Instability, Computational Efficiency and Statistical Accuracy

by   Nhat Ho, et al.

Many statistical estimators are defined as the fixed point of a data-dependent operator, with estimators based on minimizing a cost function being an important special case. The limiting performance of such estimators depends on the properties of the population-level operator in the idealized limit of infinitely many samples. We develop a general framework that yields bounds on statistical accuracy based on the interplay between the deterministic convergence rate of the algorithm at the population level, and its degree of (in)stability when applied to an empirical object based on n samples. Using this framework, we analyze both stable forms of gradient descent and some higher-order and unstable algorithms, including Newton's method and its cubic-regularized variant, as well as the EM algorithm. We provide applications of our general results to several concrete classes of models, including Gaussian mixture estimation, single-index models, and informative non-response models. We exhibit cases in which an unstable algorithm can achieve the same statistical accuracy as a stable algorithm in exponentially fewer steps—namely, with the number of iterations being reduced from polynomial to logarithmic in sample size n.



There are no comments yet.


page 1

page 2

page 3

page 4


Towards Statistical and Computational Complexities of Polyak Step Size Gradient Descent

We study the statistical and computational complexities of the Polyak st...

Statistical guarantees for the EM algorithm: From population to sample-based analysis

We develop a general framework for proving rigorous guarantees on the pe...

Sharp global convergence guarantees for iterative nonconvex optimization: A Gaussian process perspective

We consider a general class of regression models with normally distribut...

Singularity, Misspecification, and the Convergence Rate of EM

A line of recent work has characterized the behavior of the EM algorithm...

One-Step Estimation With Scaled Proximal Methods

We study statistical estimators computed using iterative optimization me...

On the Properties of Simulation-based Estimators in High Dimensions

Considering the increasing size of available data, the need for statisti...

Efficient ML Direction of Arrival Estimation assuming Unknown Sensor Noise Powers

This paper presents an efficient method for computing maximum likelihood...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.