DeepAI AI Chat
Log In Sign Up

Fitting very flexible models: Linear regression with large numbers of parameters

by   David W. Hogg, et al.

There are many uses for linear fitting; the context here is interpolation and denoising of data, as when you have calibration data and you want to fit a smooth, flexible function to those data. Or you want to fit a flexible function to de-trend a time series or normalize a spectrum. In these contexts, investigators often choose a polynomial basis, or a Fourier basis, or wavelets, or something equally general. They also choose an order, or number of basis functions to fit, and (often) some kind of regularization. We discuss how this basis-function fitting is done, with ordinary least squares and extensions thereof. We emphasize that it is often valuable to choose far more parameters than data points, despite folk rules to the contrary: Suitably regularized models with enormous numbers of parameters generalize well and make good predictions for held-out data; over-fitting is not (mainly) a problem of having too many parameters. It is even possible to take the limit of infinite parameters, at which, if the basis and regularization are chosen correctly, the least-squares fit becomes the mean of a Gaussian process. We recommend cross-validation as a good empirical method for model selection (for example, setting the number of parameters and the form of the regularization), and jackknife resampling as a good empirical method for estimating the uncertainties of the predictions made by the model. We also give advice for building stable computational implementations.


page 1

page 2

page 3

page 4


Research on Interpolation and Data Fitting: Basis and Applications

In the era of big data, we first need to manage the data, which requires...

Fitting Sparse Markov Models to Categorical Time Series Using Regularization

The major problem of fitting a higher order Markov model is the exponent...

Variable Selection Using a Smooth Information Criterion for Multi-Parameter Regression Models

Modern variable selection procedures make use of penalization methods to...

On the use of information criteria for subset selection in least squares regression

Least squares (LS) based subset selection methods are popular in linear ...

Modelling and predicting soil carbon sequestration: is current model structure fit for purpose?

Soil carbon accounting and prediction play a key role in building decisi...

GNA: new framework for statistical data analysis

We report on the status of GNA — a new framework for fitting large-scale...

Sparse Dynamic Distribution Decomposition: Efficient Integration of Trajectory and SnapshotTime Series Data

Dynamic Distribution Decomposition (DDD) was introduced in Taylor-King e...