Generalization Error Bounds for Noisy, Iterative Algorithms

01/12/2018
by   Ankit Pensia, et al.
0

In statistical learning theory, generalization error is used to quantify the degree to which a supervised machine learning algorithm may overfit to training data. Recent work [Xu and Raginsky (2017)] has established a bound on the generalization error of empirical risk minimization based on the mutual information I(S;W) between the algorithm input S and the algorithm output W, when the loss function is sub-Gaussian. We leverage these results to derive generalization error bounds for a broad class of iterative algorithms that are characterized by bounded, noisy updates with Markovian structure. Our bounds are very general and are applicable to numerous settings of interest, including stochastic gradient Langevin dynamics (SGLD) and variants of the stochastic gradient Hamiltonian Monte Carlo (SGHMC) algorithm. Furthermore, our error bounds hold for any output function computed over the path of iterates, including the last iterate of the algorithm or the average of subsets of iterates, and also allow for non-uniform sampling of data in successive updates of the algorithm.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/15/2019

Tightening Mutual Information Based Bounds on Generalization Error

A mutual information based upper bound on the generalization error of a ...
research
09/10/2023

Generalization error bounds for iterative learning algorithms with bounded updates

This paper explores the generalization characteristics of iterative lear...
research
11/16/2021

Generalization Bounds and Algorithms for Learning to Communicate over Additive Noise Channels

An additive noise channel is considered, in which the distribution of th...
research
02/28/2023

Asymptotically Optimal Generalization Error Bounds for Noisy, Iterative Algorithms

We adopt an information-theoretic framework to analyze the generalizatio...
research
02/02/2019

On Generalization Error Bounds of Noisy Gradient Methods for Non-Convex Learning

Generalization error (also known as the out-of-sample error) measures ho...
research
07/14/2018

Generalization in quasi-periodic environments

By and large the behavior of stochastic gradient is regarded as a challe...
research
07/28/2021

Characterizing the Generalization Error of Gibbs Algorithm with Symmetrized KL information

Bounding the generalization error of a supervised learning algorithm is ...

Please sign up or login with your details

Forgot password? Click here to reset