`Basic' Generalization Error Bounds for Least Squares Regression with Well-specified Models

by   Karthik Duraisamy, et al.

This note examines the behavior of generalization capabilities - as defined by out-of-sample mean squared error (MSE) - of Linear Gaussian (with a fixed design matrix) and Linear Least Squares regression. Particularly, we consider a well-specified model setting, i.e. we assume that there exists a `true' combination of model parameters within the chosen model form. While the statistical properties of Least Squares regression have been extensively studied over the past few decades - particularly with less restrictive problem statements compared to the present work - this note targets bounds that are non-asymptotic and more quantitative compared to the literature. Further, the analytical formulae for distributions and bounds (on the MSE) are directly compared to numerical experiments. Derivations are presented in a self-contained and pedagogical manner, in a way that a reader with a basic knowledge of probability and statistics can follow.



There are no comments yet.


page 1

page 2

page 3

page 4


Subsampled Optimization: Statistical Guarantees, Mean Squared Error Approximation, and Sampling Method

For optimization on large-scale data, exactly calculating its solution m...

A Characterization of Mean Squared Error for Estimator with Bagging

Bagging can significantly improve the generalization performance of unst...

Distribution-free properties of isotonic regression

It is well known that the isotonic least squares estimator is characteri...

A rigorous introduction for linear models

This note is meant to provide an introduction to linear models and the t...

On Mean Absolute Error for Deep Neural Network Based Vector-to-Vector Regression

In this paper, we exploit the properties of mean absolute error (MAE) as...

Estimation and Inference with Trees and Forests in High Dimensions

We analyze the finite sample mean squared error (MSE) performance of reg...

A model-free approach to linear least squares regression with exact probabilities

In a regression setting with observation vector y ∈ R^n and given finite...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.