Flow-Grounded Spatial-Temporal Video Prediction from Still Images

07/25/2018
by   Yijun Li, et al.
0

Existing video prediction methods mainly rely on observing multiple historical frames or focus on predicting the next one-frame. In this work, we study the problem of generating consecutive multiple future frames by observing one single still image only. We formulate the multi-frame prediction task as a multiple time step flow (multi-flow) prediction phase followed by a flow-to-frame synthesis phase. The multi-flow prediction is modeled in a variational probabilistic manner with spatial-temporal relationships learned through 3D convolutions. The flow-to-frame synthesis is modeled as a generative process in order to keep the predicted results lying closer to the manifold shape of real video sequence. Such a two-phase design prevents the model from directly looking at the high-dimensional pixel space of the frame sequence and is demonstrated to be more effective in predicting better and diverse results. Extensive experimental results on videos with different types of motion show that the proposed algorithm performs favorably against existing methods in terms of quality, diversity and human perceptual evaluation.

READ FULL TEXT

page 2

page 5

page 9

page 12

page 14

research
08/01/2017

Dual Motion GAN for Future-Flow Embedded Video Prediction

Future frame prediction in videos is a promising avenue for unsupervised...
research
11/02/2019

Quadratic video interpolation

Video interpolation is an important problem in computer vision, which he...
research
11/30/2017

Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation

Given two consecutive frames, video interpolation aims at generating int...
research
04/06/2020

Cascaded Deep Video Deblurring Using Temporal Sharpness Prior

We present a simple and effective deep convolutional neural network (CNN...
research
02/23/2020

Exploring Spatial-Temporal Multi-Frequency Analysis for High-Fidelity and Temporal-Consistency Video Prediction

Video prediction is a pixel-wise dense prediction task to infer future f...
research
03/04/2018

Improving Multi-Step Traffic Flow Prediction

In its simplest form, the traffic flow prediction problem is restricted ...
research
03/06/2023

Polar Prediction of Natural Videos

Observer motion and continuous deformations of objects and surfaces imbu...

Please sign up or login with your details

Forgot password? Click here to reset