Fast-rate PAC-Bayes Generalization Bounds via Shifted Rademacher Processes

08/20/2019
by   Jun Yang, et al.
4

The developments of Rademacher complexity and PAC-Bayesian theory have been largely independent. One exception is the PAC-Bayes theorem of Kakade, Sridharan, and Tewari (2008), which is established via Rademacher complexity theory by viewing Gibbs classifiers as linear operators. The goal of this paper is to extend this bridge between Rademacher complexity and state-of-the-art PAC-Bayesian theory. We first demonstrate that one can match the fast rate of Catoni's PAC-Bayes bounds (Catoni, 2007) using shifted Rademacher processes (Wegkamp, 2003; Lecué and Mitchell, 2012; Zhivotovskiy and Hanneke, 2018). We then derive a new fast-rate PAC-Bayes bound in terms of the "flatness" of the empirical risk surface on which the posterior concentrates. Our analysis establishes a new framework for deriving fast-rate PAC-Bayes bounds and yields new insights on PAC-Bayesian theory.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/24/2020

A Limitation of the PAC-Bayes Framework

PAC-Bayes is a useful framework for deriving generalization bounds which...
research
09/12/2022

A Note on the Efficient Evaluation of PAC-Bayes Bounds

When utilising PAC-Bayes theory for risk certification, it is usually ne...
research
06/17/2021

PAC-Bayes, MAC-Bayes and Conditional Mutual Information: Fast rate bounds that handle general VC classes

We give a novel, unified derivation of conditional PAC-Bayesian and mutu...
research
01/16/2019

A Primer on PAC-Bayesian Learning

Generalized Bayesian learning algorithms are increasingly popular in mac...
research
02/28/2012

PAC-Bayesian Generalization Bound on Confusion Matrix for Multi-Class Classification

In this work, we propose a PAC-Bayes bound for the generalization risk o...
research
10/20/2022

Tighter PAC-Bayes Generalisation Bounds by Leveraging Example Difficulty

We introduce a modified version of the excess risk, which can be used to...
research
10/19/2020

PAC^m-Bayes: Narrowing the Empirical Risk Gap in the Misspecified Bayesian Regime

While the decision-theoretic optimality of the Bayesian formalism under ...

Please sign up or login with your details

Forgot password? Click here to reset