Statistically Optimal First Order Algorithms: A Proof via Orthogonalization

by   Andrea Montanari, et al.

We consider a class of statistical estimation problems in which we are given a random data matrix X∈ℝ^n× d (and possibly some labels y∈ℝ^n) and would like to estimate a coefficient vector θ∈ℝ^d (or possibly a constant number of such vectors). Special cases include low-rank matrix estimation and regularized estimation in generalized linear models (e.g., sparse regression). First order methods proceed by iteratively multiplying current estimates by X or its transpose. Examples include gradient descent or its accelerated variants. Celentano, Montanari, Wu proved that for any constant number of iterations (matrix vector multiplications), the optimal first order algorithm is a specific approximate message passing algorithm (known as `Bayes AMP'). The error of this estimator can be characterized in the high-dimensional asymptotics n,d→∞, n/d→δ, and provides a lower bound to the estimation error of any first order algorithm. Here we present a simpler proof of the same result, and generalize it to broader classes of data distributions and of first order algorithms, including algorithms with non-separable nonlinearities. Most importantly, the new proof technique does not require to construct an equivalent tree-structured estimation problem, and is therefore susceptible of a broader range of applications.


Estimation of Low-Rank Matrices via Approximate Message Passing

Consider the problem of estimating a low-rank symmetric matrix when its ...

The estimation error of general first order methods

Modern large-scale statistical models require to estimate thousands to m...

Finite Sample Analysis of Approximate Message Passing Algorithms

Approximate message passing (AMP) refers to a class of efficient algorit...

Dense Limit of the Dawid-Skene Model for Crowdsourcing and Regions of Sub-optimality of Message Passing Algorithms

Crowdsourcing is a strategy to categorize data through the contribution ...

Phase Transitions, Optimal Errors and Optimality of Message-Passing in Generalized Linear Models

We consider generalized linear models (GLMs) where an unknown n-dimensio...

Graph-based Approximate Message Passing Iterations

Approximate-message passing (AMP) algorithms have become an important el...

Robust Regularized Low-Rank Matrix Models for Regression and Classification

While matrix variate regression models have been studied in many existin...