Jensen-Shannon Information Based Characterization of the Generalization Error of Learning Algorithms

10/23/2020
by   Gholamali Aminian, et al.
49

Generalization error bounds are critical to understanding the performance of machine learning models. In this work, we propose a new information-theoretic based generalization error upper bound applicable to supervised learning scenarios. We show that our general bound can specialize in various previous bounds. We also show that our general bound can be specialized under some conditions to a new bound involving the Jensen-Shannon information between a random variable modelling the set of training samples and another random variable modelling the set of hypotheses. We also prove that our bound can be tighter than mutual information-based bounds under some conditions.

READ FULL TEXT

page 1

page 2

page 3

page 4

01/15/2019

Tightening Mutual Information Based Bounds on Generalization Error

A mutual information based upper bound on the generalization error of a ...
02/03/2021

Information-Theoretic Bounds on the Moments of the Generalization Error of Learning Algorithms

Generalization error bounds are critical to understanding the performanc...
03/09/2019

Strengthened Information-theoretic Bounds on the Generalization Error

The following problem is considered: given a joint distribution P_XY and...
01/28/2022

Stochastic Chaining and Strengthened Information-Theoretic Generalization Bounds

We propose a new approach to apply the chaining technique in conjunction...
10/03/2021

Information-Theoretic Generalization Bounds for Iterative Semi-Supervised Learning

We consider iterative semi-supervised learning (SSL) algorithms that ite...
07/28/2021

Characterizing the Generalization Error of Gibbs Algorithm with Symmetrized KL information

Bounding the generalization error of a supervised learning algorithm is ...
01/07/2019

A New Perspective on Machine Learning: How to do Perfect Supervised Learning

In this work, we introduce the concept of bandlimiting into the theory o...