SSH: A Self-Supervised Framework for Image Harmonization

08/15/2021
by   Yifan Jiang, et al.
0

Image harmonization aims to improve the quality of image compositing by matching the "appearance" (, color tone, brightness and contrast) between foreground and background images. However, collecting large-scale annotated datasets for this task requires complex professional retouching. Instead, we propose a novel Self-Supervised Harmonization framework (SSH) that can be trained using just "free" natural images without being edited. We reformulate the image harmonization problem from a representation fusion perspective, which separately processes the foreground and background examples, to address the background occlusion issue. This framework design allows for a dual data augmentation method, where diverse [foreground, background, pseudo GT] triplets can be generated by cropping an image with perturbations using 3D color lookup tables (LUTs). In addition, we build a real-world harmonization dataset as carefully created by expert users, for evaluation and benchmarking purposes. Our results show that the proposed self-supervised method outperforms previous state-of-the-art methods in terms of reference metrics, visual quality, and subject user study. Code and dataset are available at <https://github.com/VITA-Group/SSHarmonization>.

READ FULL TEXT

page 1

page 2

page 4

page 5

page 6

page 8

research
04/26/2023

Do SSL Models Have Déjà Vu? A Case of Unintended Memorization in Self-supervised Learning

Self-supervised learning (SSL) algorithms can produce useful image repre...
research
03/01/2023

Semi-supervised Parametric Real-world Image Harmonization

Learning-based image harmonization techniques are usually trained to und...
research
12/02/2022

ObjectStitch: Generative Object Compositing

Object compositing based on 2D images is a challenging problem since it ...
research
03/31/2021

Smart Scribbles for Image Mating

Image matting is an ill-posed problem that usually requires additional u...
research
10/07/2021

Virtual Multi-Modality Self-Supervised Foreground Matting for Human-Object Interaction

Most existing human matting algorithms tried to separate pure human-only...
research
03/24/2017

AMAT: Medial Axis Transform for Natural Images

We introduce Appearance-MAT (AMAT), a generalization of the medial axis ...
research
12/13/2021

Makeup216: Logo Recognition with Adversarial Attention Representations

One of the challenges of logo recognition lies in the diversity of forms...

Please sign up or login with your details

Forgot password? Click here to reset