`Basic' Generalization Error Bounds for Least Squares Regression with Well-specified Models

09/20/2021
by   Karthik Duraisamy, et al.
0

This note examines the behavior of generalization capabilities - as defined by out-of-sample mean squared error (MSE) - of Linear Gaussian (with a fixed design matrix) and Linear Least Squares regression. Particularly, we consider a well-specified model setting, i.e. we assume that there exists a `true' combination of model parameters within the chosen model form. While the statistical properties of Least Squares regression have been extensively studied over the past few decades - particularly with less restrictive problem statements compared to the present work - this note targets bounds that are non-asymptotic and more quantitative compared to the literature. Further, the analytical formulae for distributions and bounds (on the MSE) are directly compared to numerical experiments. Derivations are presented in a self-contained and pedagogical manner, in a way that a reader with a basic knowledge of probability and statistics can follow.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

04/10/2018

Subsampled Optimization: Statistical Guarantees, Mean Squared Error Approximation, and Sampling Method

For optimization on large-scale data, exactly calculating its solution m...
08/07/2019

A Characterization of Mean Squared Error for Estimator with Bagging

Bagging can significantly improve the generalization performance of unst...
12/11/2018

Distribution-free properties of isotonic regression

It is well known that the isotonic least squares estimator is characteri...
05/10/2021

A rigorous introduction for linear models

This note is meant to provide an introduction to linear models and the t...
08/12/2020

On Mean Absolute Error for Deep Neural Network Based Vector-to-Vector Regression

In this paper, we exploit the properties of mean absolute error (MAE) as...
07/07/2020

Estimation and Inference with Trees and Forests in High Dimensions

We analyze the finite sample mean squared error (MSE) performance of reg...
07/25/2018

A model-free approach to linear least squares regression with exact probabilities

In a regression setting with observation vector y ∈ R^n and given finite...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.