HyperCon: Image-To-Video Model Transfer for Video-To-Video Translation Tasks

12/10/2019
by   Ryan Szeto, et al.
20

Video-to-video translation for super-resolution, inpainting, style transfer, etc. is more difficult than corresponding image-to-image translation tasks due to the temporal consistency problem that, if left unaddressed, results in distracting flickering effects. Although video models designed from scratch produce temporally consistent results, training them to match the vast visual knowledge captured by image models requires an intractable number of videos. To combine the benefits of image and video models, we propose an image-to-video model transfer method called Hyperconsistency (HyperCon) that transforms any well-trained image model into a temporally consistent video model without fine-tuning. HyperCon works by translating a synthetic temporally interpolated video frame-wise and then aggregating over temporally localized windows on the interpolated video. It handles both masked and unmasked inputs, enabling support for even more video-to-video tasks than prior image-to-video model transfer techniques. We demonstrate HyperCon on video style transfer and inpainting, where it performs favorably compared to prior state-of-the-art video consistency and video inpainting methods, all without training on a single stylized or incomplete video.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 3

page 6

page 8

page 11

page 12

page 13

page 15

page 16

08/01/2018

Learning Blind Video Temporal Consistency

Applying image processing algorithms independently to each frame of a vi...
04/21/2019

Automatic Temporally Coherent Video Colorization

Greyscale image colorization for applications in image restoration has s...
10/20/2021

STALP: Style Transfer with Auxiliary Limited Pairing

We present an approach to example-based stylization of images that uses ...
03/31/2021

Long-Term Temporally Consistent Unpaired Video Translation from Simulated Surgical 3D Data

Research in unpaired video translation has mainly focused on short-term ...
05/11/2022

Video-ReTime: Learning Temporally Varying Speediness for Time Remapping

We propose a method for generating a temporally remapped video that matc...
03/12/2021

Learning Long-Term Style-Preserving Blind Video Temporal Consistency

When trying to independently apply image-trained algorithms to successiv...
10/14/2021

Towards Using Clothes Style Transfer for Scenario-aware Person Video Generation

Clothes style transfer for person video generation is a challenging task...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.