Generalization Error of Generalized Linear Models in High Dimensions

05/01/2020
by   Melikasadat Emami, et al.
0

At the heart of machine learning lies the question of generalizability of learned rules over previously unseen data. While over-parameterized models based on neural networks are now ubiquitous in machine learning applications, our understanding of their generalization capabilities is incomplete. This task is made harder by the non-convexity of the underlying learning problems. We provide a general framework to characterize the asymptotic generalization error for single-layer neural networks (i.e., generalized linear models) with arbitrary non-linearities, making it applicable to regression as well as classification problems. This framework enables analyzing the effect of (i) over-parameterization and non-linearity during modeling; and (ii) choices of loss function, initialization, and regularizer during learning. Our model also captures mismatch between training and test distributions. As examples, we analyze a few special cases, namely linear regression and logistic regression. We are also able to rigorously and analytically explain the double descent phenomenon in generalized linear models.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/08/2021

Asymptotics of Ridge Regression in Convolutional Models

Understanding generalization and estimation error of estimators for simp...
research
08/25/2023

Six Lectures on Linearized Neural Networks

In these six lectures, we examine what can be learnt about the behavior ...
research
06/04/2021

Out-of-Distribution Generalization in Kernel Regression

In real word applications, data generating process for training a machin...
research
01/23/2022

A Generalized Weighted Optimization Method for Computational Learning and Inversion

The generalization capacity of various machine learning models exhibits ...
research
07/10/2023

Interpreting and generalizing deep learning in physics-based problems with functional linear models

Although deep learning has achieved remarkable success in various scient...
research
05/01/2019

Quantum Generalized Linear Models

Generalized linear models (GLM) are link function based statistical mode...
research
06/08/2020

Learning the Truth From Only One Side of the Story

Learning under one-sided feedback (i.e., where examples arrive in an onl...

Please sign up or login with your details

Forgot password? Click here to reset