Bayesian Recurrent Neural Networks

04/10/2017
by   Meire Fortunato, et al.
Google
0

In this work we explore a straightforward variational Bayes scheme for Recurrent Neural Networks. Firstly, we show that a simple adaptation of truncated backpropagation through time can yield good quality uncertainty estimates and superior regularisation at only a small extra computational cost during training. Secondly, we demonstrate how a novel kind of posterior approximation yields further improvements to the performance of Bayesian RNNs. We incorporate local gradient information into the approximate posterior to sharpen it around the current batch statistics. This technique is not exclusive to recurrent neural networks and can be applied more widely to train Bayesian neural networks. We also empirically demonstrate how Bayesian RNNs are superior to traditional RNNs on a language modelling benchmark and an image captioning task, as well as showing how each of these methods improve our model over a variety of other schemes for training them. We also introduce a new benchmark for studying uncertainty for language models so future methods can be easily compared.

READ FULL TEXT

page 7

page 11

11/23/2016

Scalable Bayesian Learning of Recurrent Neural Networks for Language Modeling

Recurrent neural networks (RNNs) have shown promising performance for la...
01/27/2019

Variational Smoothing in Recurrent Neural Network Language Models

We present a new theoretical perspective of data noising in recurrent ne...
10/05/2015

Batch Normalized Recurrent Neural Networks

Recurrent Neural Networks (RNNs) are powerful models for sequential data...
02/15/2017

Training Language Models Using Target-Propagation

While Truncated Back-Propagation through Time (BPTT) is the most popular...
12/14/2019

Predictive Precompute with Recurrent Neural Networks

In both mobile and web applications, speeding up user interface response...
07/28/2015

Training recurrent networks online without backtracking

We introduce the "NoBackTrack" algorithm to train the parameters of dyna...
03/12/2019

Application of Duration-of-Stay Storage Assignment with Deep Neural Networks under Uncertainty

Optimizing storage assignment is a central problem in warehousing. Past ...

Code Repositories

BayesianRecurrentNN

Implementation of Bayesian Recurrent Neural Networks by Fortunato et. al


view repo

Please sign up or login with your details

Forgot password? Click here to reset