The Statistics of Streaming Sparse Regression

12/13/2014
by   Jacob Steinhardt, et al.
0

We present a sparse analogue to stochastic gradient descent that is guaranteed to perform well under similar conditions to the lasso. In the linear regression setup with irrepresentable noise features, our algorithm recovers the support set of the optimal parameter vector with high probability, and achieves a statistically quasi-optimal rate of convergence of Op(k log(d)/T), where k is the sparsity of the solution, d is the number of features, and T is the number of training examples. Meanwhile, our algorithm does not require any more computational resources than stochastic gradient descent. In our experiments, we find that our method substantially out-performs existing streaming algorithms on both real and simulated data.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/06/2015

Streaming regularization parameter selection via stochastic gradient descent

We propose a framework to perform streaming covariance selection. Our ap...
research
02/12/2020

Training Two-Layer ReLU Networks with Gradient Descent is Inconsistent

We prove that two-layer (Leaky)ReLU networks initialized by e.g. the wid...
research
05/01/2021

One-pass Stochastic Gradient Descent in Overparametrized Two-layer Neural Networks

There has been a recent surge of interest in understanding the convergen...
research
07/29/2020

Truncated Linear Regression in High Dimensions

As in standard linear regression, in truncated linear regression, we are...
research
10/28/2016

Adaptive regularization for Lasso models in the context of non-stationary data streams

Large scale, streaming datasets are ubiquitous in modern machine learnin...
research
09/08/2022

Stochastic gradient descent with gradient estimator for categorical features

Categorical data are present in key areas such as health or supply chain...
research
04/21/2016

Stabilized Sparse Online Learning for Sparse Data

Stochastic gradient descent (SGD) is commonly used for optimization in l...

Please sign up or login with your details

Forgot password? Click here to reset