Sketch-to-Art: Synthesizing Stylized Art Images From Sketches

by   Bingchen Liu, et al.

We propose a new approach for synthesizing fully detailed art-stylized images from sketches. Given a sketch, with no semantic tagging, and a reference image of a specific style, the model can synthesize meaningful details with colors and textures. The model consists of three modules designed explicitly for better artistic style capturing and generation. Based on a GAN framework, a dual-masked mechanism is introduced to enforce the content constraints (from the sketch), and a feature-map transformation technique is developed to strengthen the style consistency (to the reference image). Finally, an inverse procedure of instance-normalization is proposed to disentangle the style and content information, therefore yields better synthesis performance. Experiments demonstrate a significant qualitative and quantitative boost over baselines based on previous state-of-the-art techniques, adopted for the proposed process.



There are no comments yet.


page 2

page 3

page 11

page 12

page 14

page 15

page 20

page 21


Self-Supervised Sketch-to-Image Synthesis

Imagining a colored realistic image from an arbitrarily drawn sketch is ...

Reference-Based Sketch Image Colorization using Augmented-Self Reference and Dense Semantic Correspondence

This paper tackles the automatic colorization task of a sketch image giv...

Face Sketch Synthesis with Style Transfer using Pyramid Column Feature

In this paper, we propose a novel framework based on deep neural network...

Data Incubation – Synthesizing Missing Data for Handwriting Recognition

In this paper, we demonstrate how a generative model can be used to buil...

Synthesizing Queries via Interactive Sketching

We propose a novel approach to program synthesis, focusing on synthesizi...

Dual Adversarial Inference for Text-to-Image Synthesis

Synthesizing images from a given text description involves engaging two ...

Content-Consistent Generation of Realistic Eyes with Style

Accurately labeled real-world training data can be scarce, and hence rec...

Code Repositories


The pytorch implementation of the paper: Sketch-to-Art: Synthesizing Stylized Art Images From Sketches

view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Synthesizing fully colored images from human-drawn sketches is an important problem, with several real-life applications. For example, colorizing sketches following a specified style can significantly reduce repetitive works in story-boarding. Most research works have focused on synthesizing photo-realistic images [2], or cartoonish images [32] from sketches. In this paper, we focus on rendering an image in a specific given artistic style based on a human-drawn sketch as input. The proposed approach is generic, however, what distinguish art images from other types of imagery is the variety of artistic styles that would affect how a sketch should be synthesized into a fully colored and textured image.

In the history of art, style can refer to an art movement (Renaissance style, Baroque style, Impressionism, etc.), or particular artist style (Cezanne style, Monet style, etc.), or a specific artwork style [27]. Style encompasses different formal elements of art, such as the color palette, the rendering of the contours (linear or painterly), the depth of the field (recessional or planer), the style of the brush strokes, the light (diffused or light-dark contrast), etc.

We propose a novel generation task: given an input sketch and a style, defined by a reference image, or an artist name, or a style category, we want to synthesize a fully colored and textured image in that style following the sketch, as shown in Figure 1. Previous works on synthesizing from sketches do not allow users to specify a style reference [26, 2]. We propose a new model to achieve this task, which takes the input sketch and sample reference image(s) from art-historical corpus, defining the desired style, to generate the results.

A sketch contains very sparse information, basically the main composition lines. The model has to guess the semantic composition of the scene and synthesize an image with semantic context implied from the training corpus. E.g., given a corpus of landscape art of different styles, the model needs to learn how different parts of the scene correlate with colors and texture given a choice of style. In the proposed approach, no semantic tagging is required for the sketches nor the style images used at both training and generation phases. The model implicitly infers the scene semantic.

The proposed approach is at the intersection between two different generation tasks: Sketch-to-image synthesis and Style transfer. Sketch-to-image synthesis focuses on rendering a colored image based on a sketch, where recent approaches focused on training deep learning models on a corpus of data, e.g.,

[26]. However, these approaches do not control the output based on a given style. Some recent approaches condition the generation on categories [2]. However, style cannot be treated as categories, since the transition between styles is smooth [5]. Moreover, we want to be able to finely specify style down to an individual image. Additionally, such approaches used the categories to infer content details on the rendered images, while in our case the reference image should only define the style not the content details, which has to be inferred. For example, the model should infer that certain area of the sketch is sky, trees, mountains, or grass; and infer details of these regions differently given the style image, which might not have the same semantic regions all together (see examples in Figure 1).

On the other hand, style transfer focuses on capturing the style of an image (or a group of images) and transfer it to a content image [6]. One obvious question would be what if we apply style transfer, where the content image is a sketch, wouldn’t that result in a stylized image synthesis based on that sketch? We show that such models are not applicable to synthesize stylized images from sketches because of the lack of content in the sketch, i.e., the approach need to both transfer style from the reference image and infer content based on the training corpus.

The proposed model has three novel components, which constitute the technical contributions of this paper, based on a GAN [7] infrastructure,:

Dual Mask Injection. A trainable layer that directly imposes sketch constraints on the feature maps in the convolution network, to increase content faithfulness.

Feature Map Transfer. An adaptive layer that applies a novel transformation on the style image’s feature map, extracting only the style information without the interference of the style images’ content.

Instance De-Normalization. A reverse procedure of Instance Norm [29] on the discriminator that learns to process the feature maps into two parts, to effectively disentanglement the style and content information.

Figure 2: Synthesized samples in resolution. The first column are the input sketches hand-drawn by human, and the first row are the style reference images

2 Related Work

Image-to-Image Translation: I2I aims to learn a transformation between two different domains of images. The application scenario is broad, including object transfiguration, season transfer, and photo enhancement. With the generative power of GAN [7]

, fruitful advances have been achieved in the I2I area. Pix2pix

[13] established a common framework to do the one-to-one mapping for paired images using conditional GAN. Then a series of unsupervised methods such as CycleGAN [34, 19, 4, 30] were proposed when paired data is not available. Furthermore, multi-modal I2I methods are introduced to simulate the real-world many-to-many mapping between image domains such as MUNIT and BicycleGAN [12, 35, 1, 18]. However, those I2I methods can not generate satisfying images from the coarse sketch’s domain, nor can they adequately reproduce the styles from the reference image, as will be shown in the experiments.

Neural Style Transfer: NST transfers textures and color palette from one image to another. In [6], the problem was formulated by transforming the statistics of multi-level feature maps in CNN layers in the form of a Gram Matrix. The follow-up works of NST have multiple directions, such as accelerating the transfer speed by training feed-forward networks [15, 31], and capturing multiple styles simultaneously with adaptive instance normalization [11, 29]. However, little attention has been paid on optimizing towards artistic style transfer from a sketch, nor on improving the style transfer quality regarding the fine-grained artistic patterns among the various NST methods [14].

Sketch-to-image Synthesis: Our task can be viewed from both the I2I and the NST perspectives, while it has its unique challenges. Firstly, unlike datasets such as horse-zebra or map-satellite imagery, the sketch-to-painting dataset is heavily unbalanced and one-to-many. The information in the sketch domain is ambiguous and sparse with only few lines, while in the painting’s domain is rich and diverse across all artistic styles. Secondly, “style transfer” approaches focus mainly on color palettes and “oil painting like” textures but pays limited attention to other important artistic attributes such as linear vs. painterly contours, texture boldness, and brush stroke styles. Thirdly, it is much harder for NST methods to be semantically meaningful, as the optimization procedure of NST is only between few images (usually one source and one target image). On the contrary, with a sketch-to-image model trained on a large corpus of images, learning semantics is made possible (the model can find the common coloring on different shapes and different locations across the images) and thus make the synthesized images semantically meaningful.

To our knowledge, there are few prior works close to our task. AutoPainter [20] propose to do sketch-to-image synthesis using conditional GANs, but their model is designed towards cartoon images. ScribblerGAN [26]

achieves user-guided sketch colorization but requires the user to provide localized color scribbles. SketchyGAN

[2] is the state-of-the-art approach for multi-modal sketch-to-image synthesis, focusing on photo-realistic rendering of images conditioned on object categories. Furthermore, [32, 33, 23]

accomplish the conditioned colorization with a reference image or more detailed style guidance, but they only optimize towards datasets that lack style and content variances. None of those works is specifically geared towards artistic images with the concern of capturing the significant style variances. In contrast, for example, Figure 

1, Figure 2, and Figure 9 demonstrate the ability of our model to capture the essential style patterns and generate diversely styled images at different genre. Meanwhile, our model achieves it without sacrificing the computing cost.

Figure 3: Overview of the model structure. adopts a U-Net structure [24]. It takes the features of the reference image from , runs style-conditioned image synthesis on the input sketch image , and outputs .

takes as input an image (alternatively sampled from real and generated images) and gives three outputs: a predicted style vector, a predicted sketch, and a real/fake signal of the image.

3 Methods

In this section, we provide an overview of our model and introduce the detailed training schema. The three dedicated components that boost the performance of the stylized image synthesis will also be described.

As shown in Figure 3, our model consists of three parts: a generator , a discriminator , and a separately pre-trained feature-extractor

. Either an unsupervised Auto-Encoder or a supervised DNN classifier, such as VGG

[28] can work as the feature-extractor. During the training of and , is fixed and provides multi-level feature-maps and a style vector of the reference image . The feature-maps serve as inputs for , and the style vector serves as the ground truth for . We train and under the Pix2pix [13] schema. In the synthesis phase, the input style is not limited to one reference image. We can always let extract style features from multiple images and combine them in various ways (averaging or weighted averaging) before feeding them to .

3.1 Reinforcing Content: Dual Mask Injection

In our case, the content comes in the form of a sketch, which only provides sparse compositional constraints. It is not desirable to transfer composition elements from the style image or training corpus into empty areas of the sketch that should imply textured areas. Typically when training a generator to provide images with diverse style patterns, the model tends to lose faithfulness to the content, which results in missing or excrescent shapes, objects, and ambiguous contours (especially common in NST methods). To strengthen the content faithfulness, we introduce Dual-Mask Injection layer (DMI), which directly imposes the sketch information on the intermediate feature-maps during the forward-pass in .

Given the features of a style image, in the form of the conv-layer activation , we down sample the binary input sketch to the same size of and use it as a feature mask, denoted as . The proposed DMI layer will first filter out a contour-area feature and a plain-area feature by:


where is element-wise product. For and , the DMI layer has two sets of trainable weights and biases, , that serve for a value relocation purpose, to differentiate the features around edge and plain area:


Finally, the output feature maps of DMI will be:


A real-time forward flow of the DMI layer is shown in Figure 4. Notice that, when and , the output feature will be the same as the input, and we set the weights and bias along the channel dimension so that the model can learn to impose the sketch on the feature-maps at different degrees on different channels. By imposing the sketch directly to the feature-maps, DMI layer ensures that the generated images have correct and clear contours and compositions. While DMI serves the same purpose as the masked residual layer (MRU) in SketchyGAN [2], it comes with almost zero extra computing cost, where MRU requires three more convolution layer per unit. In our experiments, our model is two times faster in training compared with SketchyGAN, while yields better results on art dataset. Moreover, the lightweight property of DMI enables our model to achieve great performance on resolution, while SketchyGAN was studied only on resolution.

Figure 4: Forward flow of Dual-Mask Injection layer, we aggregate the first three channels in feature-map as an RGB image for visualization purpose.

3.2 Disentangling Style and Content by Instance De-Normalization

According to previous studies in the conditional-GAN field, a discriminator that takes the condition as input is not as powerful as one that can actively predict the condition, and the generator usually benefits more from the latter [21, 22, 3]. In our framework, to better guide to produce more style-accurate images, is trained to adequately predict an style latent representation of the input image as well as the content sketch. Ideally, a well-disentangled discriminator can learn a style representation without the interference of the content information, and retrieve the content sketch regardless of the styles.

Several works have been done in specifically disentangling style and content [17, 16], but they only work around the generator side, using AdaIN or Gram-matrix to separate the factors. To train to effectively disentangle, we propose the Instance De-Normalization (IDN) layer. IDN takes the feature-map of an image as input, then reverses the process of Instance-Normalization [11, 29], and produces a style vector and a content feature map. In the training phrase of our model, IDN helps learn to predict accurate style-vectors and contents of the ground truth images, therefore, helps to synthesis better.

In AdaIN or IN, a stylized feature-map is calculated by:


where and calculate the variance and mean of the feature map respectively. It assumes that while the original feature map contains the content information, some external and can be collaborated with to produce a stylized feature map. The resulted feature map possesses the style information while also preserves the original content information.

Here, we reverse the process for a stylized feature map by: first predict and , () from , then separate them out from to get which carries the style-invariant content information. Formally, the IDN process is:


where is 2 conv layers. Note that unlike in AdaIN where can be directly computed from the known style feature, in IDN the ground truth style feature is unknown, thus we should not naively compute the mean of . Therefore, we use conv layers to actively predict the style information, and will reshape the output into a vector as . Finally, we concatenate and to predict the style-vector with MLP, and use conv layers on to predict the sketch. The whole IDN process can be trained end-to-end with the target style-vector and target-sketch. Unlike other disentangling methods, we separate the style and content from a structural perspective that is straightforward while maintaining effectiveness.

3.3 Reinforcing Style: Feature Map Transformation

To approach the conditional image generation task, previous methods such as MUNIT [12] and BicycleGAN [35] use a low-dimensional latent vector of extracted from some feature extractors as the conditioning factor. However, we argue that such vector representation carries limited information in terms of style details, since the artistic texture and edge styles usually lay on the fine-grained level (evidence can be found in experiment section). Therefore, it is more effective to use high-dimensional feature-maps that are closer to the image level as a conditional factor and information supplier.

Nevertheless, the low-level feature-maps usually carry both the style information and strong content information. Such content information can be problematic and is undesired. For instance, if the style image is a house while the input sketch implies a lake, we do not want any shape or illusion of a house within the lake region in the synthesized image. To get rid of the content information, while keeping the richness of the style features, we propose the Feature Map Transformation (FMT). FMT takes the input sketch , the sketch of the style image , and the feature map of the style image as inputs, and produces transformed feature-maps . only preserves the desired style features of the style image and discards its content structure. Note that is extracted using simple edge detection methods and comes from the feature-extractor .

Figure 5: Process flow of Feature-Map Transformation.

The proposed FMT is a fixed formula without parameter-training. The procedure is illustrated in Figure 5 with two steps. In step 1, we use as a mask to filter and get two sets of features, i.e., that only have the features around the contours and

with features on plain areas. Then we apply a series of max-pooling and average-pooling to this filtered yet sparse feature values to extract a

feature-map for each part, namely and . In step 2, we repeatedly fill the and into a and a with the same size of . Then we use as a mask in the same manner to filter these two feature maps, and get and that have the features of but in the shape of . Finally, we add the results to get as the output of the FMT layer. We then concatenate to its corresponding feature-maps in the generator, and process it by the next convolution layer.

The pooling operation helps collect the most distinguishable feature values along each spatial channel in , then the repeat-filling operation expands the collected global statistics, finally the masking operation makes sure the transformed feature-map will not introduce any undesired content information of the style image. Existing style-matching methods such as AdaIN [11] and Gram-matrix [6] only focus on getting higher-level statistics of the feature maps, thus have limited impacts on the detail of the generated images. In contrast, FMT provides accurate guidance to the generation of fine-grained style patterns in a straightforward manner, and has a lower computing cost compared with the aforementioned methods.

In practice, We apply FMT on the high-level feature maps ( to ). FMT contains a stack of two max-pooling and one avg-pooling layers, all with

kernel and stride of 3. The receptive field of FMT is 53x53, which is reasonable compared to the size of the feature map to get the highlighted style representations.

3.4 Objective Functions

Besides the original GAN loss, auxiliary losses are adopted to train our model. Specifically, when the input is a real image, we train with a style loss and a content loss, which minimize the MSE of the predicted style vector and sketch with the ground-truth ones respectively. Meanwhile, aims to deceive to predict the same style vector as the one extracted from and output the same sketch as . These auxiliary losses strengthen ’s ability to understand the input images and let generate more accurate style patterns while ensuring the content faithfulness. During training, we have two types of input for the model, one is the paired data, where the and are from the same image, and the other is randomly matched data, where and are not paired. Similar to Pix2pix, we have a reconstruction MSE loss on the paired data.

Formally, gives three outputs: , , , where is the predicted style vector of an image , is the predicted sketch, and

is the probability of

being a real image. Thus, the loss functions for

and are:


When the inputs are paired, the extra loss for G: , is applied. is randomly sampled real data and is its corresponding sketch, and and are randomly sampled sketches and referential style images as the input for .

4 Experiments

We first show comparisons between our model and baseline methods. Then, we specifically show the comparison to the state-of-the-art model SketchyGAN. Finally, we present the ablation studies. The code to reproduce all the experiments with detailed training configurations are included in the supplementary materials and will later be released online. An online website for people to synthesize images with their own sketches will be available soon.

Dataset: Our dataset is collected from Wikiart and consists of 55 artistic styles (e.g., impressionism, realism, etc.). We follow the sketch creation method described by [2] to get the paired sketch for each painting, and use the distance-maps from the sketches as input to train the models. We split the images into training and testing set with a ratio of . All the comparisons shown in this section were conducted on the testing set, where both the sketches and the art images were unseen to the models.

Metrics: Apart from qualitative results, we use FID [9] and a classification accuracy for the quantitative comparisons. FID is a popular image generation metric that provides a perceptual similarity between two sets of images. For our task, we generate from all the testing sketches using the same , and compute the FID between and the images from the same style. We repeat this process for all style images. Since it is relatively easy for all the models to reproduce the primary color palettes of the referred , the FID scores are commonly good even when the visual quality of is poor. Thus we employ a more direct metric which computes the style classification accuracy of

. We leverage a VGG model pre-trained on ImageNet

[25] and fine-tuned on art for style classification. Intuitively, such a model will be invariant to compositions and colors, and focus more on the artistic style patterns. We record how many of are classified correctly as the style of . This reflects how well the generator captures the style features of and translates them into . The training accuracy of the style-classification VGG is 95.1%, indicating a trustworthy performance.

Comparison to Baselines: MUNIT [12] (unsupervised) and BicycleGAN [35] (supervised) are the two state-of-the-art I2I models that are comparable to our model for their ability to do conditional image translation. SketchyGAN [2] is the latest model that is dedicated to the sketch-to-image task. Pix2pix [13] is the fundamental model for the I2I process, which SketchyGAN and our model are based on. In this section, we show comparisons between our model and the aforementioned models. We also include the results from the classical NST method by [6] as a representative of that class of methods. For SketchyGAN and Pix2pix, since they are not designed for our task, we adopt their model components but use our training schema, thus enable them to do the style-conditioned sketch-to-image synthesis.

All the tested models are trained on images with resolution due to their capacity restrictions (our model can easily upscale to , thanks to its lightweight design). We make sure all the compared models have a similar generative capacity, as they have a similar amount of weights in their respective generators, except SketchyGAN which has considerably more parameters due to its MRU layer. In terms of training time, our model is almost identical to the original Pix2pix, while other models can take up to four times training time to reach their best results.

Figure 6: Comparison to baseline models. Due to the limited information in sketches, NST and MUNIT can hardly generate meaningful images, BicycleGAN only generates images with blurry edges and fuzzy colors. Pix2pix consistently gets undesirable artifacts in multiple experiments.

Qualitative results are shown in Figure 6. Except SketchyGAN, all the other methods can hardly handle the art dataset and do not produce meaningful results. We want to point out that, models like MUNIT and BicycleGAN do work well on datasets with simple sketches, where the images share one standard shape (only cats, shoes) and are well registered with white background, without any artistic features involved (semantically diversified, different color palettes and texture). In contrast, images in art dataset are much more complicated with multiple objects and different compositions, which result in bad performance for these previous models and show the effectiveness of our proposed components.

NST MUNIT Pix2pix BicycleGAN SketchyGAN Ours
mean 6.85 7.43 0.08 7.08 0.05 6.39 0.05 5.05 0.13 4.18 0.11
Classification Score
mean 0.182 0.241 0.012 0.485 0.012 0.321 0.009 0.487 0.002 0.573 0.011

Table 1: Quantitative comparison to baseline models

The quantitative results, shown in Table 1, concur with the qualitative results. It is worth noticing that, while SketchyGAN generates comparably visual-appealing images, our model outperforms it by a large margin especially in terms of style classification score, which indicates the superiority of our model in translating the right style cues from the style image into the generated images.

Comparison to SketchyGAN: Since our task of image synthesis given sketch and style references is not possible with SketchyGAN, the comparison here is not with SketchyGAN as it is, but rather with a modified version with our training schema to suit our problem definition. While the SketchyGAN results look good from a sketch colorization perspective, they are not satisfactory from the artistic style transfer perspective compared with the given style images (eg., texture of flat area, linear vs painterly). As shown in Figure 7, SketchyGAN tends to produce only flat colors on all its generations, and has an undesired color shift compared to the reference style images. In contrast, our results provide more accurate color palette and texture restoration. Apart from the quantitative result, SketchyGAN is outperformed by our model especially in terms of the fine-grained artistic styles features, such as color flatness or fuzziness, edge sharpness, and contrast.

Figure 7: Comparison to SketchyGAN

Ablation study: We perform ablation studies to evaluate the three proposed components from a customized Pix2pix model that serves as the baseline (detailed customization can be found in the Appendix). We show both the cumulative comparisons and individual benefits for each of the components. Since IDN changes the structure of the discriminator, we add an extra reference model, one with a discriminator naively having two separate branches (noted as “with 2B”) that can similarly predict the sketch and style-vector, to validate the effectiveness of IDN.

In Table 2, the proposed components are cumulatively added to the model. FID and classification scores consistently show that each of the proposed components effectively contributes to the generation performance. Interestingly, DMI downgrades the score in both comparisons while brings better visual quality to the generated images. We hypothesis that since DMI is designed to enforce the faithfulness of given and reduce the content impact from , it makes more visually different from , leading to a worse score when the metrics are more biased on content rather than style.

baseline with DMI with FMT with 2B with IDN
mean 4.77 0.14 4.84 0.09 4.43 0.15 4.73 0.09 4.18 0.11

Classification Score
mean 0.485 0.012 0.477 0.007 0.512 0.009 0.479 0.013 0.573 0.011
Table 2: Ablation study of the proposed components

FMT and IDN bring the most significant score boost in FID and Classification, respectively. It is worth noticing how the added two branches on the discriminator (with 2B) hurt the performance and how IDN reverses that drawback and further boosts the performance. We argue that naively adding two branches collected conflicting information during training (content and style) and made it harder to train the models. In contrast, IDN neatly eliminates the conflict, thanks to its disentangling effect when processing the feature-maps, and enables the model to take advantage of both the content and style information during the backward optimization process.

Qualitative Comparison for DMI: In Figure 8, the left panel shows how DMI boosts the content faithfulness to the input sketches. In row (a), the model without DMI misses the sketch lines on the top and right portion, while DMI redeems all the lines in . In row (b), without DMI is affected by which has a flat and empty sky, leading to missing the lines on the top portion of . In contrast, the model with DMI successfully generates trees, clouds, and mountain views following the lines in . In row (c), without DMI totally messes up the shapes in the mid area in , while all the edges are correctly shaped with clear contrast in the one with DMI.

Figure 8: Qualitative comparisons of DMI, FMT and IDN

Qualitative Comparison for FMT: Figure 8 middle panel shows how FMT helps translate the style from the input style images. In row (a), FMT helps generate the correct colors and rich textures in the mountain as in . In row (b), with FMT inherits the smoothness from where the colors are fuzzy and the edges are blurry, while removing FMT leads to sharp edges and flat colors. When is flat without texture, FMT is also able to resume the correct style. Row (c) and row (d) demonstrate that when is clean and flat without fuzziness, without FMT generates undesired textures while FMT ensures the accurate style cue.

Qualitative Comparison for IDN: The right panel in Figure 8 shows how IDN helps to maintain a better color palette and generally reduce the artifacts. In row (a) and (b), there are visible artifacts in the generated images without IDN, while IDN greatly reduces such effects. In row (b) and (c), the images without IDN shows undesired green and red-brown colors that are not in the given style images, and IDN successfully removes those colors. In row (c) and (d), there are clear color-shifts in the generated images, which then been corrected in the model with IDN. More comparisons are available in the Appendix.

Figure 9: Qualitative results from our model trained on still-life genre
Figure 10: Effect of reference image vs. corpus

Qualitative Results on Other Image Domains: Even though focused on art, our model is generic and can be applied to other sketch-to-image tasks. Below we show the results of our model trained on apparel and human face data333the apparel dataset is from kaggle:, and the human face dataset is FFHQ: Figure 11 shows results on photo-realistic portraits, while Figure 12 shows results on apparels. Note that since these datasets do not have the artistic style variances that we are interested in, we do not think the power of the proposed modules, especially FMT, can be adequately reflected. However, it does show the state-of-the-art performance in general sketch-to-image tasks. Moreover, it shows the evidence that the model learns semantics from the training corpus.

Figure 11: Qualitative results from our model trained on human face photos. Note that how the glasses in the sketches are successfully rendered on the generated images even when there is no glasses in the style image; and how the moustache is correctly removed when there is moustache in the style images but is not indicated in the sketches.
Figure 12: Qualitative results from our model trained on apparel. For the three sets of images, the first column shows the style images, and the first row shows the sketches. Individually, for the right-most set, we input random art images as the style image, which the model never saw before. And the model is still able to get the correct shapes and colors.

4.1 Discussions and Future work

The model yields consistent performance when we try different input settings for the synthesis, including multiple reference style images either from the same style or the same artists. During the experiments, we also discovered some interesting behaviors and limitations that worth further research. During the training of the model, we assume that the model is able to learn some global knowledge from the whole corpus in synthesizing the images. This may contain some semantic features and more style characteristics. For example, In Figure 10-top, there is a house in the sketch but the style image has grassland with no color cues for for the house. However, the model is still able to properly colorize the house with black roof and window. Similarly in the bottom row, despite that there is no mountain in the style image, the model correctly generates the mountain with a red color tone, which is not apparent in the reference style image. Learning from the corpus can be a good thing for providing extra style cues apart from the style image, however, it may also cause conflicts against it, such as inaccurate coloring and excess shapes. It is worth study on how to balance the representation the model learns from the whole corpus and from the referential style image, and how to take advantage of the knowledge from the corpus for better generation. We direct the readers to the Appendix for more information.

5 Conclusion

We raised a novel task of generating artistic images from sketch while conditioned on style images. We approached this task with a novel sketch-to-art model. Unlike photo-realistic datasets, we highlighted and identified the unique properties of artistic images and pointed out the different challenges they possess. Respectively, we proposed methods that can effectively addressed these challenges. Our model synthesizes images with the awareness of more comprehensive artistic style attributes, which goes beyond color palettes, and for the first time, identifies the varied texture, contours, and plain area styles. Overall, our work pushes the boundary of the deep neural networks capturing and translating various artistic styles.

The three proposed components are orthogonal to previous I2I and NST works, and each of them is a plugin-and-play module that can be integrated into other frameworks. Even though focused on artistic styles, these modules also bring benefits to general tasks on multiple sketch-to-image datasets in our experiments (more results can be found in the Appendix), with their effectiveness and lightweight properties.


  • [1] Amjad Almahairi, Sai Rajeswar, Alessandro Sordoni, Philip Bachman, and Aaron Courville. Augmented cycleGAN: Learning many-to-many mappings from unpaired data. In ICML, pages 195–204, 2018.
  • [2] Wengling Chen and James Hays. Sketchygan: Towards diverse and realistic sketch to image synthesis. In CVPR, pages 9416–9425, 2018.
  • [3] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets. In NIPS, pages 2172–2180, 2016.
  • [4] Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. StarGAN: Unified generative adversarial networks for multi-domain image-to-image translation. In CVPR, pages 8789–8797, 2018.
  • [5] Ahmed Elgammal, Bingchen Liu, Diana Kim, Mohamed Elhoseiny, and Marian Mazzone. The shape of art history in the eyes of the machine. In AAAI, 2018.
  • [6] Leon A Gatys, Alexander S Ecker, and Matthias Bethge.

    Image style transfer using convolutional neural networks.

    In CVPR, pages 2414–2423, 2016.
  • [7] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, pages 2672–2680, 2014.
  • [8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770–778, 2016.
  • [9] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In NIPS, pages 6626–6637, 2017.
  • [10] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In CVPR, pages 7132–7141, 2018.
  • [11] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV, pages 1501–1510, 2017.
  • [12] Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to-image translation. In ECCV, pages 172–189, 2018.
  • [13] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.

    Image-to-image translation with conditional adversarial networks.

    In CVPR, pages 1125–1134, 2017.
  • [14] Yongcheng Jing, Yezhou Yang, Zunlei Feng, Jingwen Ye, Yizhou Yu, and Mingli Song. Neural style transfer: A review. arXiv preprint arXiv:1705.04058, 2017.
  • [15] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.

    Perceptual losses for real-time style transfer and super-resolution.

    In ECCV, pages 694–711, 2016.
  • [16] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, pages 4401–4410, 2019.
  • [17] Hadi Kazemi, Seyed Mehdi Iranmanesh, and Nasser Nasrabadi. Style and content disentanglement in generative adversarial networks. In WACV, pages 848–856, 2019.
  • [18] Hsin-Ying Lee, Hung-Yu Tseng, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. Diverse image-to-image translation via disentangled representations. In ECCV, pages 35–51, 2018.
  • [19] Ming-Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. In NIPS, pages 700–708, 2017.
  • [20] Yifan Liu, Zengchang Qin, Zhenbo Luo, and Hua Wang. Auto-painter: Cartoon image generation from sketch by using conditional generative adversarial networks. arXiv preprint arXiv:1705.01908, 2017.
  • [21] Takeru Miyato and Masanori Koyama. cGANs with projection discriminator. arXiv preprint arXiv:1802.05637, 2018.
  • [22] Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier GANs. In ICML, pages 2642–2651, 2017.
  • [23] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In CVPR, pages 2337–2346, 2019.
  • [24] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234–241, 2015.
  • [25] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 115(3):211–252, 2015.
  • [26] Patsorn Sangkloy, Jingwan Lu, Chen Fang, Fisher Yu, and James Hays. Scribbler: Controlling deep image synthesis with sketch and color. In CVPR, pages 5400–5409, 2017.
  • [27] Meyer Schapiro, Meyer Schapiro, Meyer Schapiro, and Meyer Schapiro. Theory and philosophy of art: Style, artist, and society, volume 4. George Braziller New York, 1994.
  • [28] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
  • [29] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022, 2016.
  • [30] Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image-to-image translation. In ICCV, pages 2849–2857, 2017.
  • [31] Hang Zhang and Kristin Dana. Multi-style generative network for real-time transfer. In ECCV, 2018.
  • [32] Lvmin Zhang, Yi Ji, Xin Lin, and Chunping Liu. Style transfer for anime sketches with enhanced residual u-net and auxiliary classifier gan. In ACPR, pages 506–511, 2017.
  • [33] Lvmin Zhang, Chengze Li, Tien-Tsin Wong, Yi Ji, and Chunping Liu. Two-stage sketch colorization. In SIGGRAPH Asia 2018 Technical Papers, page 261, 2018.
  • [34] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, pages 2223–2232, 2017.
  • [35] Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. In NIPS, pages 465–476, 2017.

Appendix A More Qualitative Results

Figure 13: Synthesizing with extracted sketches from paintings in testing set
Figure 14: Synthesizing with extracted sketches from paintings in testing set
Figure 15: Synthesizing with human hand-drawn sketches

Appendix B Model Structure — Attention-based Residual Block

Description: We customize the convolution structure in and based on the residual blocks proposed by [8]. Specifically, we apply a channel-wise attention following [10] on the second convolution layer in each block. To our knowledge, we are the first to adopt such attention-based layer within residual blocks. This tweak brings significant image quality boost in our task from the convolution layers used in Pix2Pix and BicycleGAN [13, 35] while maintains minimum extra computing cost. In sketch-to-image task, traditional convolution layers or residual convolutions suffer from fuzzy artifices and can hardly generate diverse colors. The proposed attention-based residual block largely improved such scenario for the baseline models. All the experimental results we present in this paper are based on this tweak. Further experiments are required to validate its effectiveness in general tasks, however, it is an orthogonal component that is beyond the discussion scope of this paper.

Figure 16: Attention-based residual block

The tweak of the attention-based residual block is illustrated in Figure 16. It consists of two convolution layers and one fully-connected layer (linear layer). It takes two input, one is the feature-map from the previous layer, and the other one is an style vector from our feature extractor . During training, the two convolution layers will compute a new feature-map , while the linear layer will take as input and output a channel-wise weight . Unlike traditional residual block which directly add back to , will provide the weights to scale the values in each channel of before the add-back operation.

Intuition: We assume that different channels in the feature-map carry the information that corresponding to different artistic styles. If we have a weight vector that can control the attendance of each channel, i.e. mute some channels’ value to zero and amplify some other channels’ value, it will make the residual block easier to learn a diversified features. Also, the extra style vector provides more information during the generation process. The generative power is therefore largely increased.

During training, the extra linear layer in the residual block introduces almost none extra computing time. However, it makes the model much faster to converge (the model is able to generate meaningful images usually after just one epoch of training), and also results in much better final performance.

Appendix C Discussions and Limitations

Figure 17: Limitations: While our model gains marked progress in synthesizing artistic images from sketches, some style patterns are hard to be captured and well-represented by the current model, e.g., the pointillism style and some styles with large patches of colors. Our model has a relatively inconsistent performance on pointillism style. In row 1 where has an intense pointy texture over the whole composition, while is trying to imitate the pointy technique around the sky area, the result shows more of flatness across the whole image. Style with large patches of color is the one on which our model has a generally unsatisfying performance. As shown in row 2, can hardly reproduce the neatly arranged color patches in , even though it achieves the correct color palette. We believe some dedicated style recognition and generation methods can be developed for these two styles.