Robust Regression via Hard Thresholding

06/08/2015
by   Kush Bhatia, et al.
0

We study the problem of Robust Least Squares Regression (RLSR) where several response variables can be adversarially corrupted. More specifically, for a data matrix X ∈ R^p x n and an underlying model w*, the response vector is generated as y = X'w* + b where b ∈ R^n is the corruption vector supported over at most C.n coordinates. Existing exact recovery results for RLSR focus solely on L1-penalty based convex formulations and impose relatively strict model assumptions such as requiring the corruptions b to be selected independently of X. In this work, we study a simple hard-thresholding algorithm called TORRENT which, under mild conditions on X, can recover w* exactly even if b corrupts the response variables in an adversarial manner, i.e. both the support and entries of b are selected adversarially after observing X and w*. Our results hold under deterministic assumptions which are satisfied if X is sampled from any sub-Gaussian distribution. Finally unlike existing results that apply only to a fixed w*, generated independently of X, our results are universal and hold for any w* ∈ R^p. Next, we propose gradient descent-based extensions of TORRENT that can scale efficiently to large scale problems, such as high dimensional sparse recovery and prove similar recovery guarantees for these extensions. Empirically we find TORRENT, and more so its extensions, offering significantly faster recovery than the state-of-the-art L1 solvers. For instance, even on moderate-sized datasets (with p = 50K) with around 40 proposed method called TORRENT-HYB is more than 20x faster than the best L1 solver.

READ FULL TEXT
research
01/24/2019

High Dimensional Robust Estimation of Sparse Models via Trimmed Hard Thresholding

We study the problem of sparsity constrained M-estimation with arbitrary...
research
10/20/2014

On Iterative Hard Thresholding Methods for High-dimensional M-Estimation

The use of M-estimators in generalized linear regression models in high ...
research
03/19/2019

Adaptive Hard Thresholding for Near-optimal Consistent Robust Regression

We study the problem of robust linear regression with response variable ...
research
10/19/2020

Robust High Dimensional Expectation Maximization Algorithm via Trimmed Hard Thresholding

In this paper, we study the problem of estimating latent variable models...
research
12/24/2022

A Bayesian Robust Regression Method for Corrupted Data Reconstruction

Because of the widespread existence of noise and data corruption, recove...
research
01/12/2013

Robust High Dimensional Sparse Regression and Matching Pursuit

We consider high dimensional sparse regression, and develop strategies a...
research
06/02/2020

Modified Hard Thresholding Pursuit with Regularization Assisted Support Identification

Hard thresholding pursuit (HTP) is a recently proposed iterative sparse ...

Please sign up or login with your details

Forgot password? Click here to reset