VAE^2: Preventing Posterior Collapse of Variational Video Predictions in the Wild

01/28/2021
by   Yizhou Zhou, et al.
2

Predicting future frames of video sequences is challenging due to the complex and stochastic nature of the problem. Video prediction methods based on variational auto-encoders (VAEs) have been a great success, but they require the training data to contain multiple possible futures for an observed video sequence. This is hard to be fulfilled when videos are captured in the wild where any given observation only has a determinate future. As a result, training a vanilla VAE model with these videos inevitably causes posterior collapse. To alleviate this problem, we propose a novel VAE structure, dabbed VAE-in-VAE or VAE^2. The key idea is to explicitly introduce stochasticity into the VAE. We treat part of the observed video sequence as a random transition state that bridges its past and future, and maximize the likelihood of a Markov Chain over the video sequence under all possible transition states. A tractable lower bound is proposed for this intractable objective function and an end-to-end optimization algorithm is designed accordingly. VAE^2 can mitigate the posterior collapse problem to a large extent, as it breaks the direct dependence between future and observation and does not directly regress the determinate future provided by the training data. We carry out experiments on a large-scale dataset called Cityscapes, which contains videos collected from a number of urban cities. Results show that VAE^2 is capable of predicting diverse futures and is more resistant to posterior collapse than the other state-of-the-art VAE-based approaches. We believe that VAE^2 is also applicable to other stochastic sequence prediction problems where training data are lack of stochasticity.

READ FULL TEXT

Authors

page 7

page 16

page 17

page 18

page 19

page 20

page 21

02/16/2019

WiSE-VAE: Wide Sample Estimator VAE

Variational Auto-encoders (VAEs) have been very successful as methods fo...
03/02/2021

Predicting Video with VQVAE

In recent years, the task of video prediction-forecasting future video g...
02/18/2021

Clockwork Variational Autoencoders

Deep learning has enabled algorithms to generate realistic images. Howev...
06/22/2020

Hierarchical Patch VAE-GAN: Generating Diverse Videos from a Single Sample

We consider the task of generating diverse and novel videos from a singl...
11/06/2019

Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse

Posterior collapse in Variational Autoencoders (VAEs) arises when the va...
10/09/2020

Deep Sequence Learning for Video Anticipation: From Discrete and Deterministic to Continuous and Stochastic

Video anticipation is the task of predicting one/multiple future represe...
08/30/2019

BooVAE: A scalable framework for continual VAE learning under boosting approach

Variational Auto Encoders (VAE) are capable of generating realistic imag...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.