Robust Linear Regression: Optimal Rates in Polynomial Time

06/29/2020
by   Ainesh Bakshi, et al.
0

We obtain a robust and computationally efficient estimator for Linear Regression that achieves statistically optimal convergence rate under mild distributional assumptions. Concretely, we assume our data is drawn from a k-hypercontractive distribution and an ϵ-fraction is adversarially corrupted. We then describe an estimator that converges to the optimal least-squares minimizer for the true distribution at a rate proportional to ϵ^2-2/k, when the noise is independent of the covariates. We note that no such estimator was known prior to our work, even with access to unbounded computation. The rate we achieve is information-theoretically optimal and thus we resolve the main open question in Klivans, Kothari and Meka [COLT'18]. Our key insight is to identify an analytic condition relating the distribution over the noise and covariates that completely characterizes the rate of convergence, regardless of the noise model. In particular, we show that when the moments of the noise and covariates are negatively-correlated, we obtain the same rate as independent noise. Further, when the condition is not satisfied, we obtain a rate proportional to ϵ^2-4/k, and again match the information-theoretic lower bound. Our central technical contribution is to algorithmically exploit independence of random variables in the "sum-of-squares" framework by formulating it as a polynomial identity.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/31/2018

Efficient Algorithms and Lower Bounds for Robust Linear Regression

We study the problem of high-dimensional linear regression in a robust m...
research
02/25/2021

Distribution-Free Robust Linear Regression

We study random design linear regression with no assumptions on the dist...
research
09/19/2020

Suboptimality of Constrained Least Squares and Improvements via Non-Linear Predictors

We study the problem of predicting as well as the best linear predictor ...
research
05/26/2023

Feature Adaptation for Sparse Linear Regression

Sparse linear regression is a central problem in high-dimensional statis...
research
10/06/2021

Least square estimators in linear regression models under negatively superadditive dependent random observations

In this article we study the asymptotic behaviour of the least square es...
research
02/22/2021

Adversarial robust weighted Huber regression

We propose a novel method to estimate the coefficients of linear regress...
research
12/06/2014

Adjusted least squares fitting of algebraic hypersurfaces

We consider the problem of fitting a set of points in Euclidean space by...

Please sign up or login with your details

Forgot password? Click here to reset