From Here to There: Video Inbetweening Using Direct 3D Convolutions

05/24/2019
by   Yunpeng Li, et al.
0

We consider the problem of generating plausible and diverse video sequences, when we are only given a start and an end frame. This task is also known as inbetweening, and it belongs to the broader area of stochastic video generation, which is generally approached by means of recurrent neural networks (RNN). In this paper, we propose instead a fully convolutional model to generate video sequences directly in the pixel domain. We first obtain a latent video representation using a stochastic fusion mechanism that learns how to incorporate information from the start and end frames. Our model learns to produce such latent representation by progressively increasing the temporal resolution, and then decode in the spatiotemporal domain using 3D convolutions. The model is trained end-to-end by minimizing an adversarial loss. Experiments on several widely-used benchmark datasets show that it is able to generate meaningful and diverse in-between video sequences, according to both quantitative and qualitative evaluations.

READ FULL TEXT

page 7

page 13

page 16

research
02/21/2018

Stochastic Video Generation with a Learned Prior

Generating video frames that accurately predict future world states is c...
research
09/01/2018

Stochastic Video Long-term Interpolation

Video interpolation is aiming to generate intermediate sequence between ...
research
04/20/2020

End-to-End Learning for Video Frame Compression with Self-Attention

One of the core components of conventional (i.e., non-learned) video cod...
research
04/05/2019

Point-to-Point Video Generation

While image manipulation achieves tremendous breakthroughs (e.g., genera...
research
12/15/2019

Brain-Inspired Inference on Missing Video Sequence

In this paper, we propose a novel end-to-end architecture that could gen...
research
06/17/2023

Fast Fourier Inception Networks for Occluded Video Prediction

Video prediction is a pixel-level task that generates future frames by e...

Please sign up or login with your details

Forgot password? Click here to reset