Fitting very flexible models: Linear regression with large numbers of parameters

01/15/2021
by   David W. Hogg, et al.
0

There are many uses for linear fitting; the context here is interpolation and denoising of data, as when you have calibration data and you want to fit a smooth, flexible function to those data. Or you want to fit a flexible function to de-trend a time series or normalize a spectrum. In these contexts, investigators often choose a polynomial basis, or a Fourier basis, or wavelets, or something equally general. They also choose an order, or number of basis functions to fit, and (often) some kind of regularization. We discuss how this basis-function fitting is done, with ordinary least squares and extensions thereof. We emphasize that it is often valuable to choose far more parameters than data points, despite folk rules to the contrary: Suitably regularized models with enormous numbers of parameters generalize well and make good predictions for held-out data; over-fitting is not (mainly) a problem of having too many parameters. It is even possible to take the limit of infinite parameters, at which, if the basis and regularization are chosen correctly, the least-squares fit becomes the mean of a Gaussian process. We recommend cross-validation as a good empirical method for model selection (for example, setting the number of parameters and the form of the regularization), and jackknife resampling as a good empirical method for estimating the uncertainties of the predictions made by the model. We also give advice for building stable computational implementations.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/05/2018

Cross validation residuals for generalised least squares and other correlated data models

Cross validation residuals are well known for the ordinary least squares...
research
08/25/2022

Research on Interpolation and Data Fitting: Basis and Applications

In the era of big data, we first need to manage the data, which requires...
research
02/11/2022

Fitting Sparse Markov Models to Categorical Time Series Using Regularization

The major problem of fitting a higher order Markov model is the exponent...
research
11/22/2019

On the use of information criteria for subset selection in least squares regression

Least squares (LS) based subset selection methods are popular in linear ...
research
05/11/2021

Modelling and predicting soil carbon sequestration: is current model structure fit for purpose?

Soil carbon accounting and prediction play a key role in building decisi...
research
03/13/2019

GNA: new framework for statistical data analysis

We report on the status of GNA — a new framework for fitting large-scale...
research
06/09/2020

Sparse Dynamic Distribution Decomposition: Efficient Integration of Trajectory and SnapshotTime Series Data

Dynamic Distribution Decomposition (DDD) was introduced in Taylor-King e...

Please sign up or login with your details

Forgot password? Click here to reset