Fast global convergence of gradient methods for high-dimensional statistical recovery

04/25/2011
by   Alekh Agarwal, et al.
0

Many statistical M-estimators are based on convex optimization problems formed by the combination of a data-dependent loss function with a norm-based regularizer. We analyze the convergence rates of projected gradient and composite gradient methods for solving such problems, working within a high-dimensional framework that allows the data dimension to grow with (and possibly exceed) the sample size . This high-dimensional structure precludes the usual global assumptions---namely, strong convexity and smoothness conditions---that underlie much of classical optimization analysis. We define appropriately restricted versions of these conditions, and show that they are satisfied with high probability for various statistical models. Under these conditions, our theory guarantees that projected gradient descent has a globally geometric rate of convergence up to the statistical precision of the model, meaning the typical distance between the true unknown parameter θ^* and an optimal solution θ̂. This result is substantially sharper than previous convergence results, which yielded sublinear convergence, or linear convergence only up to the noise level. Our analysis applies to a wide range of M-estimators and statistical models, including sparse linear regression using Lasso (ℓ_1-regularized regression); group Lasso for block sparsity; log-linear models with regularization; low-rank matrix recovery using nuclear norm regularization; and matrix decomposition. Overall, our analysis reveals interesting connections between statistical precision and computational efficiency in high-dimensional estimation.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/21/2022

High-Dimensional Inference over Networks: Linear Convergence and Statistical Guarantees

We study sparse linear regression over a network of agents, modeled as a...
research
02/24/2017

Computationally Efficient Robust Estimation of Sparse Functionals

Many conventional statistical procedures are extremely sensitive to seem...
research
09/16/2011

High-dimensional regression with noisy and missing data: Provable guarantees with nonconvexity

Although the standard formulations of prediction problems involve fully-...
research
03/22/2016

Localized Lasso for High-Dimensional Regression

We introduce the localized Lasso, which is suited for learning models th...
research
04/12/2022

High-dimensional nonconvex lasso-type M-estimators

This paper proposes a theory for ℓ_1-norm penalized high-dimensional M-e...
research
11/15/2016

Errors-in-variables models with dependent measurements

Suppose that we observe y ∈R^n and X ∈R^n × m in the following errors-in...
research
07/14/2023

Performance of ℓ_1 Regularization for Sparse Convex Optimization

Despite widespread adoption in practice, guarantees for the LASSO and Gr...

Please sign up or login with your details

Forgot password? Click here to reset