Predictive Complexity Priors

06/18/2020
by   Eric Nalisnick, et al.
0

Specifying a Bayesian prior is notoriously difficult for complex models such as neural networks. Reasoning about parameters is made challenging by the high-dimensionality and over-parameterization of the space. Priors that seem benign and uninformative can have unintuitive and detrimental effects on a model's predictions. For this reason, we propose predictive complexity priors: a functional prior that is defined by comparing the model's predictions to those of a reference function. Although originally defined on the model outputs, we transfer the prior to the model parameters via a change of variables. The traditional Bayesian workflow can then proceed as usual. We apply our predictive complexity prior to modern machine learning tasks such as reasoning over neural network depth and sharing of statistical strength for few-shot learning.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/13/2021

Enriched standard conjugate priors and the right invariant prior for Wishart distributions

We investigate Bayesian predictions for Wishart distributions by using t...
research
11/25/2020

All You Need is a Good Functional Prior for Bayesian Deep Learning

The Bayesian treatment of neural networks dictates that a prior distribu...
research
12/20/2020

Dimension-robust Function Space MCMC With Neural Network Priors

This paper introduces a new prior on functions spaces which scales more ...
research
04/04/2017

Learning Approximately Objective Priors

Informative Bayesian priors are often difficult to elicit, and when this...
research
05/15/2019

Output-Constrained Bayesian Neural Networks

Bayesian neural network (BNN) priors are defined in parameter space, mak...
research
07/22/2018

The Median Probability Model and Correlated Variables

The median probability model (MPM) Barbieri and Berger (2004) is defined...
research
01/03/2021

Learning optimal Bayesian prior probabilities from data

Noninformative uniform priors are staples of Bayesian inference, especia...

Please sign up or login with your details

Forgot password? Click here to reset