Flexible Diffusion Modeling of Long Videos

05/23/2022
by   William Harvey, et al.
6

We present a framework for video modeling based on denoising diffusion probabilistic models that produces long-duration video completions in a variety of realistic environments. We introduce a generative model that can at test-time sample any arbitrary subset of video frames conditioned on any other subset and present an architecture adapted for this purpose. Doing so allows us to efficiently compare and optimize a variety of schedules for the order in which frames in a long video are sampled and use selective sparse and long-range conditioning on previously sampled frames. We demonstrate improved video modeling over prior work on a number of datasets and sample temporally coherent videos over 25 minutes in length. We additionally release a new video modeling dataset and semantically meaningful metrics based on videos generated in the CARLA self-driving car simulator.

READ FULL TEXT

page 2

page 6

page 15

page 16

page 17

research
10/05/2022

Temporally Consistent Video Transformer for Long-Term Video Prediction

Generating long, temporally consistent video remains an open challenge i...
research
04/07/2022

Video Diffusion Models

Generating temporally coherent high fidelity video is an important miles...
research
02/15/2023

Video Probabilistic Diffusion Models in Projected Latent Space

Despite the remarkable progress in deep generative models, synthesizing ...
research
03/16/2022

Diffusion Probabilistic Modeling for Video Generation

Denoising diffusion probabilistic models are a promising new class of ge...
research
07/05/2018

Consistent Generative Query Networks

Stochastic video prediction is usually framed as an extrapolation proble...
research
08/21/2018

Deep Video-Based Performance Cloning

We present a new video-based performance cloning technique. After traini...
research
02/16/2003

Analysis and Interface for Instructional Video

We present a new method for segmenting, and a new user interface for ind...

Please sign up or login with your details

Forgot password? Click here to reset