Function Space Particle Optimization for Bayesian Neural Networks

02/26/2019
by   Ziyu Wang, et al.
24

While Bayesian neural networks (BNNs) have drawn increasing attention, their posterior inference remains challenging, due to the high-dimensional and over-parameterized nature. To address this issue, several highly flexible and scalable variational inference procedures based on the idea of particle optimization have been proposed. These methods directly optimize a set of particles to approximate the target posterior. However, their application to BNNs often yields sub-optimal performance, as such methods have a particular failure mode on over-parameterized models. In this paper, we propose to solve this issue by performing particle optimization directly in the space of regression functions. We demonstrate through extensive experiments that our method successfully overcomes this issue, and outperforms strong baselines in a variety of tasks including prediction, defense against adversarial examples, and reinforcement learning.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/29/2018

A Unified Particle-Optimization Framework for Scalable Bayesian Sampling

There has been recent interest in developing scalable Bayesian sampling ...
research
05/27/2019

Walsh-Hadamard Variational Inference for Bayesian Deep Learning

Over-parameterized models, such as DeepNets and ConvNets, form a class o...
research
03/25/2020

Scalable Variational Gaussian Process Regression Networks

Gaussian process regression networks (GPRN) are powerful Bayesian models...
research
01/26/2023

Coin Sampling: Gradient-Based Bayesian Inference without Learning Rates

In recent years, particle-based variational inference (ParVI) methods su...
research
02/28/2023

Particle-based Online Bayesian Sampling

Online optimization has gained increasing interest due to its capability...
research
03/04/2023

Progressive Bayesian Particle Flows based on Optimal Transport Map Sequences

We propose a method for optimal Bayesian filtering with deterministic pa...
research
06/09/2021

Loss function based second-order Jensen inequality and its application to particle variational inference

Bayesian model averaging, obtained as the expectation of a likelihood fu...

Please sign up or login with your details

Forgot password? Click here to reset