Deep Plastic Surgery: Robust and Controllable Image Editing with Human-Drawn Sketches

01/09/2020
by   Shuai Yang, et al.
10

Sketch-based image editing aims to synthesize and modify photos based on the structural information provided by the human-drawn sketches. Since sketches are difficult to collect, previous methods mainly use edge maps instead of sketches to train models (referred to as edge-based models). However, sketches display great structural discrepancy with edge maps, thus failing edge-based models. Moreover, sketches often demonstrate huge variety among different users, demanding even higher generalizability and robustness for the editing model to work. In this paper, we propose Deep Plastic Surgery, a novel, robust and controllable image editing framework that allows users to interactively edit images using hand-drawn sketch inputs. We present a sketch refinement strategy, as inspired by the coarse-to-fine drawing process of the artists, which we show can help our model well adapt to casual and varied sketches without the need for real sketch training data. Our model further provides a refinement level control parameter that enables users to flexibly define how "reliable" the input sketch should be considered for the final output, balancing between sketch faithfulness and output verisimilitude (as the two goals might contradict if the input sketch is drawn poorly). To achieve the multi-level refinement, we introduce a style-based module for level conditioning, which allows adaptive feature representations for different levels in a singe network. Extensive experimental results demonstrate the superiority of our approach in improving the visual quality and user controllablity of image editing over the state-of-the-art methods.

READ FULL TEXT

page 1

page 6

page 7

page 8

page 13

page 14

page 15

page 16

research
04/06/2023

SketchFFusion: Sketch-guided image editing with diffusion model

Sketch-guided image editing aims to achieve local fine-tuning of the ima...
research
03/01/2023

Sketch2Cloth: Sketch-based 3D Garment Generation with Unsigned Distance Fields

3D model reconstruction from a single image has achieved great progress ...
research
06/18/2020

Multi-Density Sketch-to-Image Translation Network

Sketch-to-image (S2I) translation plays an important role in image synth...
research
06/09/2023

Sketch Beautification: Learning Part Beautification and Structure Refinement for Sketches of Man-made Objects

We present a novel freehand sketch beautification method, which takes as...
research
03/23/2021

DeFLOCNet: Deep Image Editing via Flexible Low-level Controls

User-intended visual content fills the hole regions of an input image in...
research
04/01/2021

Sketch2Mesh: Reconstructing and Editing 3D Shapes from Sketches

Reconstructing 3D shape from 2D sketches has long been an open problem b...
research
02/16/2018

A complete hand-drawn sketch vectorization framework

Vectorizing hand-drawn sketches is a challenging task, which is of param...

Please sign up or login with your details

Forgot password? Click here to reset