Free-form Video Inpainting with 3D Gated Convolution and Temporal PatchGAN

04/23/2019
by   Ya-Liang Chang, et al.
0

Free-form video inpainting is a very challenging task that could be widely used for video editing such as text removal. Existing patch-based methods could not handle non-repetitive structures such as faces, while directly applying image-based inpainting models to videos will result in temporal inconsistency (see http://bit.ly/2Fu1n6b). In this paper, we introduce a deep learn-ing based free-form video inpainting model, with proposed 3D gated convolutions to tackle the uncertainty of free-form masks and a novel Temporal PatchGAN loss to enhance temporal consistency. In addition, we collect videos and design a free-form mask generation algorithm to build the free-form video inpainting (FVI) dataset for training and evaluation of video inpainting models. We demonstrate the benefits of these components and experiments on both the FaceForensics and our FVI dataset suggest that our method is superior to existing ones.

READ FULL TEXT

page 1

page 3

page 7

page 8

research
07/02/2019

Learnable Gated Temporal Shift Module for Deep Video Inpainting

How to efficiently utilize temporal information to recover videos in a c...
research
06/10/2018

Free-Form Image Inpainting with Gated Convolution

We present a novel deep learning based image inpainting system to comple...
research
05/11/2021

The DEVIL is in the Details: A Diagnostic Evaluation Benchmark for Video Inpainting

Quantitative evaluation has increased dramatically among recent video in...
research
03/18/2015

Video Inpainting of Complex Scenes

We propose an automatic video inpainting algorithm which relies on the o...
research
12/17/2019

Dim the Lights! – Low-Rank Prior Temporal Data for Specular-Free Video Recovery

The appearance of an object is significantly affected by the illuminatio...
research
04/14/2019

VORNet: Spatio-temporally Consistent Video Inpainting for Object Removal

Video object removal is a challenging task in video processing that ofte...
research
12/16/2022

Free-form 3D Scene Inpainting with Dual-stream GAN

Nowadays, the need for user editing in a 3D scene has rapidly increased ...

Please sign up or login with your details

Forgot password? Click here to reset