Bayesian Recurrent Neural Networks

by   Meire Fortunato, et al.

In this work we explore a straightforward variational Bayes scheme for Recurrent Neural Networks. Firstly, we show that a simple adaptation of truncated backpropagation through time can yield good quality uncertainty estimates and superior regularisation at only a small extra computational cost during training. Secondly, we demonstrate how a novel kind of posterior approximation yields further improvements to the performance of Bayesian RNNs. We incorporate local gradient information into the approximate posterior to sharpen it around the current batch statistics. This technique is not exclusive to recurrent neural networks and can be applied more widely to train Bayesian neural networks. We also empirically demonstrate how Bayesian RNNs are superior to traditional RNNs on a language modelling benchmark and an image captioning task, as well as showing how each of these methods improve our model over a variety of other schemes for training them. We also introduce a new benchmark for studying uncertainty for language models so future methods can be easily compared.


page 7

page 11


Scalable Bayesian Learning of Recurrent Neural Networks for Language Modeling

Recurrent neural networks (RNNs) have shown promising performance for la...

Variational Smoothing in Recurrent Neural Network Language Models

We present a new theoretical perspective of data noising in recurrent ne...

Batch Normalized Recurrent Neural Networks

Recurrent Neural Networks (RNNs) are powerful models for sequential data...

Training Language Models Using Target-Propagation

While Truncated Back-Propagation through Time (BPTT) is the most popular...

Predictive Precompute with Recurrent Neural Networks

In both mobile and web applications, speeding up user interface response...

Training recurrent networks online without backtracking

We introduce the "NoBackTrack" algorithm to train the parameters of dyna...

Application of Duration-of-Stay Storage Assignment with Deep Neural Networks under Uncertainty

Optimizing storage assignment is a central problem in warehousing. Past ...

Code Repositories


Implementation of Bayesian Recurrent Neural Networks by Fortunato et. al

view repo

Please sign up or login with your details

Forgot password? Click here to reset