Learning Wake-Sleep Recurrent Attention Models

09/22/2015
by   Jimmy Ba, et al.
0

Despite their success, convolutional neural networks are computationally expensive because they must examine all image locations. Stochastic attention-based models have been shown to improve computational efficiency at test time, but they remain difficult to train because of intractable posterior inference and high variance in the stochastic gradient estimates. Borrowing techniques from the literature on training deep generative models, we present the Wake-Sleep Recurrent Attention Model, a method for training stochastic attention networks which improves posterior inference and which reduces the variability in the stochastic gradients. We show that our method can greatly speed up the training time for stochastic attention networks in the domains of image classification and caption generation.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
03/30/2016

Deep Networks with Stochastic Depth

Very deep convolutional networks with hundreds of layers have led to sig...
research
06/11/2014

Reweighted Wake-Sleep

Training deep directed graphical models with many hidden variables and p...
research
07/10/2018

Latent Alignment and Variational Attention

Neural attention has become central to many state-of-the-art models in n...
research
06/13/2019

Near-Optimal Glimpse Sequences for Improved Hard Attention Neural Network Training

We introduce the use of Bayesian optimal experimental design techniques ...
research
08/15/2022

Applying Regularized Schrödinger-Bridge-Based Stochastic Process in Generative Modeling

Compared to the existing function-based models in deep generative modeli...
research
03/18/2017

A wake-sleep algorithm for recurrent, spiking neural networks

We investigate a recently proposed model for cortical computation which ...
research
07/06/2020

Learning to learn generative programs with Memoised Wake-Sleep

We study a class of neuro-symbolic generative models in which neural net...

Please sign up or login with your details

Forgot password? Click here to reset