Log In Sign Up

An Information-Theoretic Perspective on Overfitting and Underfitting

by   Daniel Bashir, et al.

We present an information-theoretic framework for understanding overfitting and underfitting in machine learning and prove the formal undecidability of determining whether an arbitrary classification algorithm will overfit a dataset. Measuring algorithm capacity via the information transferred from datasets to models, we consider mismatches between algorithm capacities and datasets to provide a signature for when a model can overfit or underfit a dataset. We present results upper-bounding algorithm capacity, establish its relationship to quantities in the algorithmic search framework for machine learning, and relate our work to recent information-theoretic approaches to generalization.


page 1

page 2

page 3

page 4


Undecidability of Underfitting in Learning Algorithms

Using recent machine learning results that present an information-theore...

Irregular Repetition Slotted ALOHA in an Information-Theoretic Setting

An information-theoretic approach to irregular repetition slotted ALOHA ...

An information-theoretic approach to the analysis of location and co-location patterns

We propose a statistical framework to quantify location and co-location ...

Redundancy of Hidden Layers in Deep Learning: An Information Perspective

Although the deep structure guarantees the powerful expressivity of deep...

Efficient Information Theoretic Clustering on Discrete Lattices

We consider the problem of clustering data that reside on discrete, low ...

Big Variates: Visualizing and identifying key variables in a multivariate world

Big Data involves both a large number of events but also many variables....

An Information-Theoretic Perspective on the Relationship Between Fairness and Accuracy

Our goal is to understand the so-called trade-off between fairness and a...