Fast and Robust Online Inference with Stochastic Gradient Descent via Random Scaling

06/06/2021
by   Sokbae Lee, et al.
0

We develop a new method of online inference for a vector of parameters estimated by the Polyak-Ruppert averaging procedure of stochastic gradient descent (SGD) algorithms. We leverage insights from time series regression in econometrics and construct asymptotically pivotal statistics via random scaling. Our approach is fully operational with online data and is rigorously underpinned by a functional central limit theorem. Our proposed inference method has a couple of key advantages over the existing methods. First, the test statistic is computed in an online fashion with only SGD iterates and the critical values can be obtained without any resampling methods, thereby allowing for efficient implementation suitable for massive online data. Second, there is no need to estimate the asymptotic variance and our inference method is shown to be robust to changes in the tuning parameters for SGD algorithms in simulation experiments with synthetic data.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/13/2023

Weighted Averaged Stochastic Gradient Descent: Asymptotic Normality and Optimality

Stochastic Gradient Descent (SGD) is one of the simplest and most popula...
research
09/29/2022

Fast Inference for Quantile Regression with Tens of Millions of Observations

While applications of big data analytics have brought many new opportuni...
research
04/27/2023

Fairness Uncertainty Quantification: How certain are you that the model is fair?

Fairness-aware machine learning has garnered significant attention in re...
research
05/21/2021

Online Statistical Inference for Parameters Estimation with Linear-Equality Constraints

Stochastic gradient descent (SGD) and projected stochastic gradient desc...
research
02/13/2018

Statistical Inference for Online Learning and Stochastic Approximation via Hierarchical Incremental Gradient Descent

Stochastic gradient descent (SGD) is an immensely popular approach for o...
research
05/21/2017

Statistical inference using SGD

We present a novel method for frequentist statistical inference in M-est...
research
07/01/2020

Online Robust Regression via SGD on the l1 loss

We consider the robust linear regression problem in the online setting w...

Please sign up or login with your details

Forgot password? Click here to reset