Replica analysis of overfitting in generalized linear models

by   ACC Coolen, et al.

Nearly all statistical inference methods were developed for the regime where the number N of data samples is much larger than the data dimension p. Inference protocols such as maximum likelihood (ML) or maximum a posteriori probability (MAP) are unreliable if p=O(N), due to overfitting. This limitation has for many disciplines with increasingly high-dimensional data become a serious bottleneck. We recently showed that in Cox regression for time-to-event data the overfitting errors are not just noise but take mostly the form of a bias, and how with the replica method from statistical physics once can model and predict this bias and the noise statistics. Here we extend our approach to arbitrary generalized linear regression models (GLM), with possibly correlated covariates. We analyse overfitting in ML/MAP inference without having to specify data types or regression models, relying only on the GLM form, and derive generic order parameter equations for the case of L2 priors. Second, we derive the probabilistic relationship between true and inferred regression coefficients in GLMs, and show that, for the relevant hyperparameter scaling and correlated covariates, the L2 regularization causes a predictable direction change of the coefficient vector. Our results, illustrated by application to linear, logistic, and Cox regression, enable one to correct ML and MAP inferences in GLMs systematically for overfitting bias, and thus extend their applicability into the hitherto forbidden regime p=O(N).



There are no comments yet.


page 1

page 2

page 3

page 4


Analysis of overfitting in the regularized Cox model

The Cox proportional hazards model is ubiquitous in the analysis of time...

The Existence of Maximum Likelihood Estimate in High-Dimensional Generalized Linear Models with Binary Responses

Motivated by recent works on the high-dimensional logistic regression, w...

Detecting non-causal artifacts in multivariate linear regression models

We consider linear models where d potential causes X_1,...,X_d are corre...

Concentration Based Inference in High Dimensional Generalized Regression Models (I: Statistical Guarantees)

We develop simple and non-asymptotically justified methods for hypothesi...

Bias Reduction as a Remedy to the Consequences of Infinite Estimates in Poisson and Tobit Regression

Data separation is a well-studied phenomenon that can cause problems in ...

A Simple Correction Procedure for High-Dimensional General Linear Models with Measurement Error

We consider high-dimensional generalized linear models when the covariat...

Optimal regularizations for data generation with probabilistic graphical models

Understanding the role of regularization is a central question in Statis...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.