Bayesian Dark Knowledge

06/14/2015
by   Anoop Korattikara, et al.
0

We consider the problem of Bayesian parameter estimation for deep neural networks, which is important in problem settings where we may have little data, and/ or where we need accurate posterior predictive densities, e.g., for applications involving bandits or active learning. One simple approach to this is to use online Monte Carlo methods, such as SGLD (stochastic gradient Langevin dynamics). Unfortunately, such a method needs to store many copies of the parameters (which wastes memory), and needs to make predictions using many versions of the model (which wastes time). We describe a method for "distilling" a Monte Carlo approximation to the posterior predictive density into a more compact form, namely a single deep neural network. We compare to two very recent approaches to Bayesian neural networks, namely an approach based on expectation propagation [Hernandez-Lobato and Adams, 2015] and an approach based on variational Bayes [Blundell et al., 2015]. Our method performs better than both of these, is much simpler to implement, and uses less computation at test time.

READ FULL TEXT
research
12/15/2022

Bayesian posterior approximation with stochastic ensembles

We introduce ensembles of stochastic neural networks to approximate the ...
research
09/06/2022

Bayesian Neural Network Inference via Implicit Models and the Posterior Predictive Distribution

We propose a novel approach to perform approximate Bayesian inference in...
research
12/31/2015

Distributed Bayesian Learning with Stochastic Natural-gradient Expectation Propagation and the Posterior Server

This paper makes two contributions to Bayesian machine learning algorith...
research
06/04/2019

Assessing the Robustness of Bayesian Dark Knowledge to Posterior Uncertainty

Bayesian Dark Knowledge is a method for compressing the posterior predic...
research
12/11/2019

Bayesian Variational Autoencoders for Unsupervised Out-of-Distribution Detection

Despite their successes, deep neural networks still make unreliable pred...
research
05/16/2020

Generalized Bayesian Posterior Expectation Distillation for Deep Neural Networks

In this paper, we present a general framework for distilling expectation...
research
06/27/2018

Adversarial Distillation of Bayesian Neural Network Posteriors

Bayesian neural networks (BNNs) allow us to reason about uncertainty in ...

Please sign up or login with your details

Forgot password? Click here to reset