Taylor Swift: Taylor Driven Temporal Modeling for Swift Future Frame Prediction

While recurrent neural networks (RNNs) demonstrate outstanding capabilities in future video frame prediction, they model dynamics in a discrete time space and sequentially go through all frames until the desired future temporal step is reached. RNNs are therefore prone to accumulate the error as the number of future frames increases. In contrast, partial differential equations (PDEs) model physical phenomena like dynamics in continuous time space, however, current PDE-based approaches discretize the PDEs using e.g., the forward Euler method. In this work, we therefore propose to approximate the motion in a video by a continuous function using the Taylor series. To this end, we introduce TayloSwiftNet, a novel convolutional neural network that learns to estimate the higher order terms of the Taylor series for a given input video. TayloSwiftNet can swiftly predict any desired future frame in just one forward pass and change the temporal resolution on-the-fly. The experimental results on various datasets demonstrate the superiority of our model.

READ FULL TEXT

page 6

page 8

research
06/16/2020

Learning continuous-time PDEs from sparse data with graph neural networks

The behavior of many dynamical systems follow complex, yet still unknown...
research
04/06/2022

PAGP: A physics-assisted Gaussian process framework with active learning for forward and inverse problems of partial differential equations

In this work, a Gaussian process regression(GPR) model incorporated with...
research
02/10/2014

Modeling sequential data using higher-order relational features and predictive training

Bi-linear feature learning models, like the gated autoencoder, were prop...
research
09/04/2021

Multiscale and Nonlocal Learning for PDEs using Densely Connected RNNs

Learning time-dependent partial differential equations (PDEs) that gover...
research
03/16/2022

Backpropagation through Time and Space: Learning Numerical Methods with Multi-Agent Reinforcement Learning

We introduce Backpropagation Through Time and Space (BPTTS), a method fo...
research
10/16/2020

Vid-ODE: Continuous-Time Video Generation with Neural Ordinary Differential Equation

Video generation models often operate under the assumption of fixed fram...
research
03/03/2020

Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Prediction

Leveraging physical knowledge described by partial differential equation...

Please sign up or login with your details

Forgot password? Click here to reset