An Information-Theoretic Perspective on Overfitting and Underfitting

10/12/2020
by   Daniel Bashir, et al.
0

We present an information-theoretic framework for understanding overfitting and underfitting in machine learning and prove the formal undecidability of determining whether an arbitrary classification algorithm will overfit a dataset. Measuring algorithm capacity via the information transferred from datasets to models, we consider mismatches between algorithm capacities and datasets to provide a signature for when a model can overfit or underfit a dataset. We present results upper-bounding algorithm capacity, establish its relationship to quantities in the algorithmic search framework for machine learning, and relate our work to recent information-theoretic approaches to generalization.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

02/04/2021

Undecidability of Underfitting in Learning Algorithms

Using recent machine learning results that present an information-theore...
06/22/2021

A Practical Unified Notation for Information-Theoretic Quantities in ML

Information theory is of importance to machine learning, but the notatio...
04/22/2020

An information-theoretic approach to the analysis of location and co-location patterns

We propose a statistical framework to quantify location and co-location ...
09/19/2020

Redundancy of Hidden Layers in Deep Learning: An Information Perspective

Although the deep structure guarantees the powerful expressivity of deep...
10/26/2013

Efficient Information Theoretic Clustering on Discrete Lattices

We consider the problem of clustering data that reside on discrete, low ...
06/10/2019

Big Variates: Visualizing and identifying key variables in a multivariate world

Big Data involves both a large number of events but also many variables....
06/07/2021

An Information-theoretic Approach to Distribution Shifts

Safely deploying machine learning models to the real world is often a ch...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.