PAC-Bayes, MAC-Bayes and Conditional Mutual Information: Fast rate bounds that handle general VC classes

06/17/2021
by   Peter Grünwald, et al.
0

We give a novel, unified derivation of conditional PAC-Bayesian and mutual information (MI) generalization bounds. We derive conditional MI bounds as an instance, with special choice of prior, of conditional MAC-Bayesian (Mean Approximately Correct) bounds, itself derived from conditional PAC-Bayesian bounds, where `conditional' means that one can use priors conditioned on a joint training and ghost sample. This allows us to get nontrivial PAC-Bayes and MI-style bounds for general VC classes, something recently shown to be impossible with standard PAC-Bayesian/MI bounds. Second, it allows us to get faster rates of order O ((KL/n)^γ) for γ > 1/2 if a Bernstein condition holds and for exp-concave losses (with γ=1), which is impossible with both standard PAC-Bayes generalization and MI bounds. Our work extends the recent work by Steinke and Zakynthinou [2020] who handle MI with VC but neither PAC-Bayes nor fast rates, the recent work of Hellström and Durisi [2020] who extend the latter to the PAC-Bayes setting via a unifying exponential inequality, and Mhammedi et al. [2019] who initiated fast rate PAC-Bayes generalization error bounds but handle neither MI nor general VC classes.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/20/2019

Fast-rate PAC-Bayes Generalization Bounds via Shifted Rademacher Processes

The developments of Rademacher complexity and PAC-Bayesian theory have b...
research
10/21/2021

User-friendly introduction to PAC-Bayes bounds

Aggregated predictors are obtained by making a set of basic predictors v...
research
10/10/2019

PAC-Bayesian Contrastive Unsupervised Representation Learning

Contrastive unsupervised representation learning (CURL) is the state-of-...
research
03/03/2022

Robust PAC^m: Training Ensemble Models Under Model Misspecification and Outliers

Standard Bayesian learning is known to have suboptimal generalization ca...
research
05/31/2019

PAC-Bayes Un-Expected Bernstein Inequality

We present a new PAC-Bayesian generalization bound. Standard bounds cont...
research
09/29/2021

PAC-Bayes Information Bottleneck

Information bottleneck (IB) depicts a trade-off between the accuracy and...
research
09/08/2023

Generalization Bounds: Perspectives from Information Theory and PAC-Bayes

A fundamental question in theoretical machine learning is generalization...

Please sign up or login with your details

Forgot password? Click here to reset