Random design analysis of ridge regression

06/13/2011
by   Daniel Hsu, et al.
0

This work gives a simultaneous analysis of both the ordinary least squares estimator and the ridge regression estimator in the random design setting under mild assumptions on the covariate/response distributions. In particular, the analysis provides sharp results on the "out-of-sample" prediction error, as opposed to the "in-sample" (fixed design) error. The analysis also reveals the effect of errors in the estimated covariance structure, as well as the effect of modeling errors, neither of which effects are present in the fixed design setting. The proofs of the main results are based on a simple decomposition lemma combined with concentration inequalities for random vectors and matrices.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/16/2022

An elementary analysis of ridge regression with random design

In this note, we provide an elementary analysis of the prediction error ...
research
05/29/2016

Singular ridge regression with homoscedastic residuals: generalization error with estimated parameters

This paper characterizes the conditional distribution properties of the ...
research
09/02/2020

Simultaneous inference for Berkson errors-in-variables regression under fixed design

In various applications of regression analysis, in addition to errors in...
research
05/04/2011

A Risk Comparison of Ordinary Least Squares vs Ridge Regression

We compare the risk of ridge regression to a simple variant of ordinary ...
research
05/26/2023

Error Bounds for Learning with Vector-Valued Random Features

This paper provides a comprehensive error analysis of learning with vect...
research
11/28/2019

Finite impulse response models: A non-asymptotic analysis of the least squares estimator

We consider a finite impulse response system with centered independent s...
research
05/03/2021

Ridge Regularized Estimation of VAR Models for Inference and Sieve Approximation

Developments in statistical learning have fueled the analysis of high-di...

Please sign up or login with your details

Forgot password? Click here to reset