PriorityCut: Occlusion-guided Regularization for Warp-based Image Animation

by   Wai Ting Cheung, et al.

Image animation generates a video of a source image following the motion of a driving video. State-of-the-art self-supervised image animation approaches warp the source image according to the motion of the driving video and recover the warping artifacts by inpainting. These approaches mostly use vanilla convolution for inpainting, and vanilla convolution does not distinguish between valid and invalid pixels. As a result, visual artifacts are still noticeable after inpainting. CutMix is a state-of-the-art regularization strategy that cuts and mixes patches of images and is widely studied in different computer vision tasks. Among the remaining computer vision tasks, warp-based image animation is one of the fields that the effects of CutMix have yet to be studied. This paper first presents a preliminary study on the effects of CutMix on warp-based image animation. We observed in our study that CutMix helps improve only pixel values, but disturbs the spatial relationships between pixels. Based on such observation, we propose PriorityCut, a novel augmentation approach that uses the top-k percent occluded pixels of the foreground to regularize warp-based image animation. By leveraging the domain knowledge in warp-based image animation, PriorityCut significantly reduces the warping artifacts in state-of-the-art warp-based image animation models on diverse datasets.


page 2

page 4

page 6

page 14

page 15

page 16


Multi-domain CT metal artifacts reduction using partial convolution based inpainting

Recent CT Metal Artifacts Reduction (MAR) methods are often based on ima...

Image Inpainting for Irregular Holes Using Partial Convolutions

Existing deep learning based image inpainting methods use a standard con...

Free-Form Image Inpainting with Gated Convolution

We present a novel deep learning based image inpainting system to comple...

ItoV: Efficiently Adapting Deep Learning-based Image Watermarking to Video Watermarking

Robust watermarking tries to conceal information within a cover image/vi...

Implicit Warping for Animation with Image Sets

We present a new implicit warping framework for image animation using se...

Semantic bottleneck for computer vision tasks

This paper introduces a novel method for the representation of images th...

Cylin-Painting: Seamless 360° Panoramic Image Outpainting and Beyond with Cylinder-Style Convolutions

Image outpainting gains increasing attention since it can generate the c...

Please sign up or login with your details

Forgot password? Click here to reset