Deep Neural Networks as Point Estimates for Deep Gaussian Processes

05/10/2021
by   Vincent Dutordoir, et al.
23

Deep Gaussian processes (DGPs) have struggled for relevance in applications due to the challenges and cost associated with Bayesian inference. In this paper we propose a sparse variational approximation for DGPs for which the approximate posterior mean has the same mathematical structure as a Deep Neural Network (DNN). We make the forward pass through a DGP equivalent to a ReLU DNN by finding an interdomain transformation that represents the GP posterior mean as a sum of ReLU basis functions. This unification enables the initialisation and training of the DGP as a neural network, leveraging the well established practice in the deep learning community, and so greatly aiding the inference task. The experiments demonstrate improved accuracy and faster training compared to current DGP methods, while retaining favourable predictive uncertainties.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 10

page 11

06/05/2019

Approximate Inference Turns Deep Networks into Gaussian Processes

Deep neural networks (DNN) and Gaussian processes (GP) are two powerful ...
05/17/2020

Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes

Variational inference is a popular approach to reason about uncertainty ...
10/12/2019

On the expected behaviour of noise regularised deep neural networks as Gaussian processes

Recent work has established the equivalence between deep neural networks...
11/19/2020

Application of Deep Learning-based Interpolation Methods to Nearshore Bathymetry

Nearshore bathymetry, the topography of the ocean floor in coastal zones...
06/10/2018

Building Bayesian Neural Networks with Blocks: On Structure, Interpretability and Uncertainty

We provide simple schemes to build Bayesian Neural Networks (BNNs), bloc...
04/12/2021

GPflux: A Library for Deep Gaussian Processes

We introduce GPflux, a Python library for Bayesian deep learning with a ...
10/10/2019

Rate Optimal Variational Bayesian Inference for Sparse DNN

Sparse deep neural network (DNN) has drawn much attention in recent stud...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.