Vid-ODE: Continuous-Time Video Generation with Neural Ordinary Differential Equation

10/16/2020
by   Sunghyun Park, et al.
0

Video generation models often operate under the assumption of fixed frame rates, which leads to suboptimal performance when it comes to handling flexible frame rates (e.g., increasing the frame rate of more dynamic portion of the video as well as handling missing video frames). To resolve the restricted nature of existing video generation models' ability to handle arbitrary timesteps, we propose continuous-time video generation by combining neural ODE (Vid-ODE) with pixel-level video processing techniques. Using ODE-ConvGRU as an encoder, a convolutional version of the recently proposed neural ODE, which enables us to learn continuous-time dynamics, Vid-ODE can learn the spatio-temporal dynamics of input videos of flexible frame rates. The decoder integrates the learned dynamics function to synthesize video frames at any given timesteps, where the pixel-level composition technique is used to maintain the sharpness of individual frames. With extensive experiments on four real-world video datasets, we verify that the proposed Vid-ODE outperforms state-of-the-art approaches under various video generation settings, both within the trained time range (interpolation) and beyond the range (extrapolation). To the best of our knowledge, Vid-ODE is the first work successfully performing continuous-time video generation using real-world videos.

READ FULL TEXT

page 6

page 14

page 15

page 17

page 18

page 19

page 20

page 21

research
12/21/2021

Continuous-Time Video Generation via Learning Motion Dynamics with Neural ODE

In order to perform unconditional video generation, we must learn the di...
research
06/25/2017

Decomposing Motion and Content for Natural Video Sequence Prediction

We propose a deep neural network for the prediction of future frames in ...
research
10/23/2022

Towards Real-Time Text2Video via CLIP-Guided, Pixel-Level Optimization

We introduce an approach to generating videos based on a series of given...
research
08/31/2020

ALANET: Adaptive Latent Attention Network forJoint Video Deblurring and Interpolation

Existing works address the problem of generating high frame-rate sharp v...
research
07/31/2023

Continuous-Time Channel Prediction Based on Tensor Neural Ordinary Differential Equation

Channel prediction is critical to address the channel aging issue in mob...
research
04/10/2022

Learning Pixel-Level Distinctions for Video Highlight Detection

The goal of video highlight detection is to select the most attractive s...
research
10/27/2021

Taylor Swift: Taylor Driven Temporal Modeling for Swift Future Frame Prediction

While recurrent neural networks (RNNs) demonstrate outstanding capabilit...

Please sign up or login with your details

Forgot password? Click here to reset