Deep multi-scale video prediction beyond mean square error

11/17/2015
by   Michael Mathieu, et al.
0

Learning to predict future images from a video sequence involves the construction of an internal representation that models the image evolution accurately, and therefore, to some degree, its content and dynamics. This is why pixel-space video prediction may be viewed as a promising avenue for unsupervised feature learning. In addition, while optical flow has been a very studied problem in computer vision for a long time, future frame prediction is rarely approached. Still, many vision applications could benefit from the knowledge of the next frames of videos, that does not require the complexity of tracking every pixel trajectories. In this work, we train a convolutional network to generate future frames given an input sequence. To deal with the inherently blurry predictions obtained from the standard Mean Squared Error (MSE) loss function, we propose three different and complementary feature learning strategies: a multi-scale architecture, an adversarial training method, and an image gradient difference loss function. We compare our predictions to different published results based on recurrent neural networks on the UCF101 dataset

READ FULL TEXT

page 8

page 9

page 12

page 13

research
11/27/2018

Deep Learned Frame Prediction for Video Compression

Motion compensation is one of the most essential methods for any video c...
research
12/07/2017

Multi-Scale Video Frame-Synthesis Network with Transitive Consistency Loss

Traditional approaches to interpolating/extrapolating frames in a video ...
research
11/16/2017

Frame Interpolation with Multi-Scale Deep Loss Functions and Generative Adversarial Networks

Frame interpolation attempts to synthesise intermediate frames given one...
research
11/19/2015

Unsupervised Learning of Visual Structure using Predictive Generative Networks

The ability to predict future states of the environment is a central pil...
research
04/12/2017

Predictive-Corrective Networks for Action Detection

While deep feature learning has revolutionized techniques for static-ima...
research
06/07/2019

Detecting the Starting Frame of Actions in Video

To understand causal relationships between events in the world, it is us...
research
05/23/2022

Deep-learning-based prediction of nanoparticle phase transitions during in situ transmission electron microscopy

We develop the machine learning capability to predict a time sequence of...

Please sign up or login with your details

Forgot password? Click here to reset