On the use of information criteria for subset selection in least squares regression

by   Sen Tian, et al.

Least squares (LS) based subset selection methods are popular in linear regression modeling when the number of predictors is less than the number of observations. Best subset selection (BS) is known to be NP hard and has a computational cost that grows exponentially with the number of predictors. Forward stepwise selection (FS) is a greedy heuristic for BS. Both methods rely on cross-validation (CV) in order to select the subset size k, which requires fitting the procedures multiple times and results in a selected k that is random across replications. Compared to CV, information criteria only require fitting the procedures once, and we show that for LS-based methods they can result in better predictive performance while providing a non-random choice of k. However, information criteria require knowledge of the effective degrees of freedom for the fitting procedure, which is generally not available analytically for complex methods. In this paper, we propose a novel LS-based method, the best orthogonalized subset selection (BOSS) method, which performs BS upon an orthogonalized basis of ordered predictors. Assuming orthogonal predictors, we build a connection between BS and its Lagrangian formulation (i.e., minimization of the residual sum of squares plus the product of a regularization parameter and k), and based on this connection introduce a heuristic degrees of freedom (hdf) for BOSS that can be estimated via an analytically-based expression. We show in both simulations and real data analysis that BOSS using the Kullback-Leibler based information criterion AICc-hdf has the strongest performance of all of the LS-based methods considered and is competitive with regularization methods, with the computational effort of a single ordinary LS fit.



There are no comments yet.


page 1

page 2

page 3

page 4


Efficient algorithm to select tuning parameters in sparse regression modeling with regularization

In sparse regression modeling via regularization such as the lasso, it i...

Selection of Regression Models under Linear Restrictions for Fixed and Random Designs

Many important modeling tasks in linear regression, including variable s...

Fitting very flexible models: Linear regression with large numbers of parameters

There are many uses for linear fitting; the context here is interpolatio...

Selection consistency of Lasso-based procedures for misspecified high-dimensional binary model and random regressors

We consider selection of random predictors for high-dimensional regressi...

Conditional Uncorrelation and Efficient Non-approximate Subset Selection in Sparse Regression

Given m d-dimensional responsors and n d-dimensional predictors, sparse ...

High-dimensional regression with potential prior information on variable importance

There are a variety of settings where vague prior information may be ava...

Scalable algorithms for semiparametric accelerated failure time models in high dimensions

Semiparametric accelerated failure time (AFT) models are a useful altern...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.