Towards Unified Keyframe Propagation Models

05/19/2022
by   Patrick Esser, et al.
7

Many video editing tasks such as rotoscoping or object removal require the propagation of context across frames. While transformers and other attention-based approaches that aggregate features globally have demonstrated great success at propagating object masks from keyframes to the whole video, they struggle to propagate high-frequency details such as textures faithfully. We hypothesize that this is due to an inherent bias of global attention towards low-frequency features. To overcome this limitation, we present a two-stream approach, where high-frequency features interact locally and low-frequency features interact globally. The global interaction stream remains robust in difficult situations such as large camera motions, where explicit alignment fails. The local interaction stream propagates high-frequency details through deformable feature aggregation and, informed by the global interaction stream, learns to detect and correct errors of the deformation field. We evaluate our two-stream approach for inpainting tasks, where experiments show that it improves both the propagation of features within a single frame as required for image inpainting, as well as their propagation from keyframes to target frames. Applied to video inpainting, our approach leads to 44 FID and LPIPS scores. Code at https://github.com/runwayml/guided-inpainting

READ FULL TEXT

page 2

page 3

page 8

page 9

page 10

page 11

page 12

page 13

research
05/30/2019

Align-and-Attend Network for Globally and Locally Coherent Video Inpainting

We propose a novel feed-forward network for video inpainting. We use a s...
research
12/03/2020

Image inpainting using frequency domain priors

In this paper, we present a novel image inpainting technique using frequ...
research
07/15/2023

Both Spatial and Frequency Cues Contribute to High-Fidelity Image Inpainting

Deep generative approaches have obtained great success in image inpainti...
research
07/18/2018

An Attention-Based Approach for Single Image Super Resolution

The main challenge of single image super resolution (SISR) is the recove...
research
01/26/2021

Deep Video Inpainting Detection

This paper studies video inpainting detection, which localizes an inpain...
research
09/07/2023

ProPainter: Improving Propagation and Transformer for Video Inpainting

Flow-based propagation and spatiotemporal Transformer are two mainstream...
research
07/01/2023

Unsupervised Coordinate-Based Video Denoising

In this paper, we introduce a novel unsupervised video denoising deep le...

Please sign up or login with your details

Forgot password? Click here to reset