A note relating ridge regression and OLS p-values to preconditioned sparse penalized regression

11/26/2014
by   Karl Rohe, et al.
0

When the design matrix has orthonormal columns, "soft thresholding" the ordinary least squares (OLS) solution produces the Lasso solution [Tibshirani, 1996]. If one uses the Puffer preconditioned Lasso [Jia and Rohe, 2012], then this result generalizes from orthonormal designs to full rank designs (Theorem 1). Theorem 2 refines the Puffer preconditioner to make the Lasso select the same model as removing the elements of the OLS solution with the largest p-values. Using a generalized Puffer preconditioner, Theorem 3 relates ridge regression to the preconditioned Lasso; this result is for the high dimensional setting, p > n. Where the standard Lasso is akin to forward selection [Efron et al., 2004], Theorems 1, 2, and 3 suggest that the preconditioned Lasso is more akin to backward elimination. These results hold for sparse penalties beyond l1; for a broad class of sparse and non-convex techniques (e.g. SCAD and MC+), the results hold for all local minima.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/15/2018

On the Complexity of the Weighted Fussed Lasso

The solution path of the 1D fused lasso for an n-dimensional input is pi...
research
01/15/2018

On the Complexity of the Weighted Fused Lasso

The solution path of the 1D fused lasso for an n-dimensional input is pi...
research
06/09/2020

On Coresets For Regularized Regression

We study the effect of norm based regularization on the size of coresets...
research
07/18/2020

Local Convergence of an AMP Variant to the LASSO Solution in Finite Dimensions

A common sparse linear regression formulation is the l1 regularized leas...
research
05/18/2020

The Trimmed Lasso: Sparse Recovery Guarantees and Practical Optimization by the Generalized Soft-Min Penalty

We present a new approach to solve the sparse approximation or best subs...
research
01/10/2014

Lasso and equivalent quadratic penalized models

The least absolute shrinkage and selection operator (lasso) and ridge re...
research
09/21/2012

A Note on the SPICE Method

In this article, we analyze the SPICE method developed in [1], and estab...

Please sign up or login with your details

Forgot password? Click here to reset