Log In Sign Up

MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motions

by   Haixu Wu, et al.

This paper tackles video prediction from a new dimension of predicting spacetime-varying motions that are incessantly changing across both space and time. Prior methods mainly capture the temporal state transitions but overlook the complex spatiotemporal variations of the motion itself, making them difficult to adapt to ever-changing motions. We observe that physical world motions can be decomposed into transient variation and motion trend, while the latter can be regarded as the accumulation of previous motions. Thus, simultaneously capturing the transient variation and the motion trend is the key to make spacetime-varying motions more predictable. Based on these observations, we propose the MotionRNN framework, which can capture the complex variations within motions and adapt to spacetime-varying scenarios. MotionRNN has two main contributions. The first is that we design the MotionGRU unit, which can model the transient variation and motion trend in a unified way. The second is that we apply the MotionGRU to RNN-based predictive models and indicate a new flexible video prediction architecture with a Motion Highway that can significantly improve the ability to predict changeable motions and avoid motion vanishing for stacked multiple-layer predictive models. With high flexibility, this framework can adapt to a series of models for deterministic spatiotemporal prediction. Our MotionRNN can yield significant improvements on three challenging benchmarks for video prediction with spacetime-varying motions.


page 1

page 6

page 7

page 8

page 11

page 12


Spatiotemporal Co-attention Recurrent Neural Networks for Human-Skeleton Motion Prediction

Human motion prediction aims to generate future motions based on the obs...

A Causal Convolutional Neural Network for Motion Modeling and Synthesis

We propose a novel deep generative model based on causal convolutions fo...

Weakly-supervised Action Transition Learning for Stochastic Human Motion Prediction

We introduce the task of action-driven stochastic human motion predictio...

PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning

We present PredRNN++, an improved recurrent network for video predictive...

Landmark-Guided Elastic Shape Analysis of Human Character Motions

Motions of virtual characters in movies or video games are typically gen...

DeepMag: Source Specific Motion Magnification Using Gradient Ascent

Many important physical phenomena involve subtle signals that are diffic...