DeepAI AI Chat
Log In Sign Up

Generalization Error of Generalized Linear Models in High Dimensions

by   Melikasadat Emami, et al.

At the heart of machine learning lies the question of generalizability of learned rules over previously unseen data. While over-parameterized models based on neural networks are now ubiquitous in machine learning applications, our understanding of their generalization capabilities is incomplete. This task is made harder by the non-convexity of the underlying learning problems. We provide a general framework to characterize the asymptotic generalization error for single-layer neural networks (i.e., generalized linear models) with arbitrary non-linearities, making it applicable to regression as well as classification problems. This framework enables analyzing the effect of (i) over-parameterization and non-linearity during modeling; and (ii) choices of loss function, initialization, and regularizer during learning. Our model also captures mismatch between training and test distributions. As examples, we analyze a few special cases, namely linear regression and logistic regression. We are also able to rigorously and analytically explain the double descent phenomenon in generalized linear models.


page 1

page 2

page 3

page 4


Asymptotics of Ridge Regression in Convolutional Models

Understanding generalization and estimation error of estimators for simp...

Out-of-Distribution Generalization in Kernel Regression

In real word applications, data generating process for training a machin...

Learning how to explain neural networks: PatternNet and PatternAttribution

DeConvNet, Guided BackProp, LRP, were invented to better understand deep...

A Generalized Weighted Optimization Method for Computational Learning and Inversion

The generalization capacity of various machine learning models exhibits ...

On Generalization of Adaptive Methods for Over-parameterized Linear Regression

Over-parameterization and adaptive methods have played a crucial role in...

Quantum Generalized Linear Models

Generalized linear models (GLM) are link function based statistical mode...

Generalization for multiclass classification with overparameterized linear models

Via an overparameterized linear model with Gaussian features, we provide...