Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

07/09/2016
by   Tianfan Xue, et al.
0

We study the problem of synthesizing a number of likely future frames from a single input image. In contrast to traditional methods, which have tackled this problem in a deterministic or non-parametric way, we propose a novel approach that models future frames in a probabilistic manner. Our probabilistic model makes it possible for us to sample and synthesize many possible future frames from a single input image. Future frame synthesis is challenging, as it involves low- and high-level image and motion understanding. We propose a novel network structure, namely a Cross Convolutional Network to aid in synthesizing future frames; this network structure encodes image and motion information as feature maps and convolutional kernels, respectively. In experiments, our model performs well on synthetic data, such as 2D shapes and animated game sprites, as well as on real-wold videos. We also show that our model can be applied to tasks such as visual analogy-making, and present an analysis of the learned network representations.

READ FULL TEXT

page 2

page 6

page 8

page 9

page 10

research
07/24/2018

Visual Dynamics: Stochastic Future Generation via Layered Cross Convolutional Networks

We study the problem of synthesizing a number of likely future frames fr...
research
02/10/2022

FILM: Frame Interpolation for Large Motion

We present a frame interpolation algorithm that synthesizes multiple int...
research
12/05/2018

Video Synthesis from a Single Image and Motion Stroke

In this paper, we propose a new method to automatically generate a video...
research
12/25/2018

Motion Selective Prediction for Video Frame Synthesis

Existing conditional video prediction approaches train a network from la...
research
08/10/2020

Deep Sketch-guided Cartoon Video Synthesis

We propose a novel framework to produce cartoon videos by fetching the c...
research
01/14/2018

Innovative Non-parametric Texture Synthesis via Patch Permutations

In this work, we present a non-parametric texture synthesis algorithm ca...
research
05/15/2018

Topological Eulerian Synthesis of Slow Motion Periodic Videos

We consider the problem of taking a video that is comprised of multiple ...

Please sign up or login with your details

Forgot password? Click here to reset