Log In Sign Up

Information bottleneck theory of high-dimensional regression: relevancy, efficiency and optimality

by   Vudtiwat Ngampruetikorn, et al.

Avoiding overfitting is a central challenge in machine learning, yet many large neural networks readily achieve zero training loss. This puzzling contradiction necessitates new approaches to the study of overfitting. Here we quantify overfitting via residual information, defined as the bits in fitted models that encode noise in training data. Information efficient learning algorithms minimize residual information while maximizing the relevant bits, which are predictive of the unknown generative models. We solve this optimization to obtain the information content of optimal algorithms for a linear regression problem and compare it to that of randomized ridge regression. Our results demonstrate the fundamental tradeoff between residual and relevant information and characterize the relative information efficiency of randomized regression with respect to optimal algorithms. Finally, using results from random matrix theory, we reveal the information complexity of learning a linear map in high dimensions and unveil information-theoretic analogs of double and multiple descent phenomena.


Statistical Mechanics of High-Dimensional Inference

To model modern large-scale datasets, we need efficient algorithms to in...

Undecidability of Underfitting in Learning Algorithms

Using recent machine learning results that present an information-theore...

Adversarial Regression. Generative Adversarial Networks for Non-Linear Regression: Theory and Assessment

Adversarial Regression is a proposition to perform high dimensional non-...

User Modelling for Avoiding Overfitting in Interactive Knowledge Elicitation for Prediction

In human-in-the-loop machine learning, the user provides information bey...

Characterizing information loss in a chaotic double pendulum with the Information Bottleneck

A hallmark of chaotic dynamics is the loss of information with time. Alt...

The Implicit Bias of Benign Overfitting

The phenomenon of benign overfitting, where a predictor perfectly fits n...

A novel residual whitening based training to avoid overfitting

In this paper we demonstrate that training models to minimize the autoco...