DeepAI
Log In Sign Up

An Information-Theoretic Perspective on Overfitting and Underfitting

10/12/2020
by   Daniel Bashir, et al.
0

We present an information-theoretic framework for understanding overfitting and underfitting in machine learning and prove the formal undecidability of determining whether an arbitrary classification algorithm will overfit a dataset. Measuring algorithm capacity via the information transferred from datasets to models, we consider mismatches between algorithm capacities and datasets to provide a signature for when a model can overfit or underfit a dataset. We present results upper-bounding algorithm capacity, establish its relationship to quantities in the algorithmic search framework for machine learning, and relate our work to recent information-theoretic approaches to generalization.

READ FULL TEXT

page 1

page 2

page 3

page 4

02/04/2021

Undecidability of Underfitting in Learning Algorithms

Using recent machine learning results that present an information-theore...
07/13/2022

Irregular Repetition Slotted ALOHA in an Information-Theoretic Setting

An information-theoretic approach to irregular repetition slotted ALOHA ...
04/22/2020

An information-theoretic approach to the analysis of location and co-location patterns

We propose a statistical framework to quantify location and co-location ...
09/19/2020

Redundancy of Hidden Layers in Deep Learning: An Information Perspective

Although the deep structure guarantees the powerful expressivity of deep...
10/26/2013

Efficient Information Theoretic Clustering on Discrete Lattices

We consider the problem of clustering data that reside on discrete, low ...
06/10/2019

Big Variates: Visualizing and identifying key variables in a multivariate world

Big Data involves both a large number of events but also many variables....
10/17/2019

An Information-Theoretic Perspective on the Relationship Between Fairness and Accuracy

Our goal is to understand the so-called trade-off between fairness and a...