Continuous conditional video synthesis by neural processes

10/11/2022
by   Xi Ye, et al.
0

We propose a unified model for multiple conditional video synthesis tasks, including video prediction and video frame interpolation. We show that conditional video synthesis can be formulated as a neural process, which maps input spatio-temporal coordinates to target pixel values given context spatio-temporal coordinates and pixels values. Specifically, we feed an implicit neural representations of coordinates into a Transformer-based non-autoregressive conditional video synthesis model. Our task-specific models outperform previous work for video interpolation on multiple datasets and reach a competitive performance with the state-of-the-art models for video prediction. Importantly, the model is able to interpolate or predict with an arbitrary high frame rate, i.e., continuous synthesis. Our source code is available at https://github.com/NPVS/NPVS.

READ FULL TEXT

page 6

page 7

page 12

page 13

page 14

page 15

page 16

research
03/29/2022

VPTR: Efficient Transformers for Video Prediction

In this paper, we propose a new Transformer block for video future frame...
research
08/12/2023

ModelScope Text-to-Video Technical Report

This paper introduces ModelScopeT2V, a text-to-video synthesis model tha...
research
10/04/2022

A Perceptual Quality Metric for Video Frame Interpolation

Research on video frame interpolation has made significant progress in r...
research
06/19/2023

Practical Equivariances via Relational Conditional Neural Processes

Conditional Neural Processes (CNPs) are a class of metalearning models p...
research
06/17/2022

Video Shadow Detection via Spatio-Temporal Interpolation Consistency Training

It is challenging to annotate large-scale datasets for supervised video ...
research
12/12/2022

Video Prediction by Efficient Transformers

Video prediction is a challenging computer vision task that has a wide r...
research
05/10/2021

Stochastic Image-to-Video Synthesis using cINNs

Video understanding calls for a model to learn the characteristic interp...

Please sign up or login with your details

Forgot password? Click here to reset