PAC^m-Bayes: Narrowing the Empirical Risk Gap in the Misspecified Bayesian Regime

by   Warren R. Morningstar, et al.

While the decision-theoretic optimality of the Bayesian formalism under correct model specification is well-known (Berger 2013), the Bayesian case becomes less clear under model misspecification (Grunwald 2017; Ramamoorthi 2015; Fushiki 2005). To formally understand the consequences of Bayesian misspecification, this work examines the relationship between posterior predictive risk and its sensitivity to correct model assumptions, i.e., choice of likelihood and prior. We present the multisample PAC^m-Bayes risk. This risk is justified by theoretical analysis based on PAC-Bayes as well as empirical study on a number of toy problems. The PAC^m-Bayes risk is appealing in that it entails direct minimization of the Monte-Carlo approximated posterior predictive risk yet recovers both the Bayesian formalism as well as the MLE in its limits. Our work is heavily influenced by Masegosa (2019); our contributions are to align training and generalization risks while offering a tighter bound which empirically performs at least as well and sometimes much better.



page 1

page 2

page 3

page 4


Fast-rate PAC-Bayes Generalization Bounds via Shifted Rademacher Processes

The developments of Rademacher complexity and PAC-Bayesian theory have b...

PAC-Bayes Bounds on Variational Tempered Posteriors for Markov Models

Datasets displaying temporal dependencies abound in science and engineer...

Learning under Model Misspecification: Applications to Variational and Ensemble methods

This paper provides a novel theoretical analysis of the problem of learn...

Learning Partially Known Stochastic Dynamics with Empirical PAC Bayes

We propose a novel scheme for fitting heavily parameterized non-linear s...

Bayesian Prior Networks with PAC Training

We propose to train Bayesian Neural Networks (BNNs) by empirical Bayes a...

A PAC-Bayes Analysis of Adversarial Robustness

We propose the first general PAC-Bayesian generalization bounds for adve...

Learning from i.i.d. data under model miss-specification

This paper introduces a new approach to learning from i.i.d. data under ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.