Compressed Sparse Linear Regression
share
High-dimensional sparse linear regression is a basic problem in machine learning and statistics. Consider a linear model y = Xθ^ + w, where y ∈R^n is the vector of observations, X ∈R^n × d is the covariate matrix and w ∈R^n is an unknown noise vector. In many applications, the linear regression model is high-dimensional in nature, meaning that the number of observations n may be substantially smaller than the number of covariates d. In these cases, it is common to assume that θ^ is sparse, and the goal in sparse linear regression is to estimate this sparse θ^, given (X,y). In this paper, we study a variant of the traditional sparse linear regression problem where each of the n covariate vectors in R^d are individually projected by a random linear transformation to R^m with m ≪ d. Such transformations are commonly applied in practice for computational savings in resources such as storage space, transmission bandwidth, and processing time. Our main result shows that one can estimate θ^ with a low ℓ_2-error, even with access to only these projected covariate vectors, under some mild assumptions on the problem instance. Our approach is based on solving a variant of the popular Lasso optimization problem. While the conditions (such as the restricted eigenvalue condition on X) for success of a Lasso formulation in estimating θ^ are well-understood, we investigate conditions under which this variant of Lasso estimates θ^. As a simple consequence, our approach also provides a new way for estimating θ^ in the traditional sparse linear regression problem setting, which operates (even) under a weaker assumption on the design matrix than previously known, albeit achieving a weaker convergence bound.
READ FULL TEXT