Implementation of our paper, Text2Colors: Guiding Image Colorization through Text-Driven Palette Generation
In this paper, we propose a novel approach to generate multiple color palettes that reflect the semantics of input text and then colorize a given grayscale image according to the generated color palette. In contrast to existing approaches, our model can understand rich text, whether it is a single word, a phrase, or a sentence, and generate multiple possible palettes from it. To achieve this task, we introduce our manually curated dataset called Palette-and-Text (PAT), which consists of 10,183 pairs of text and its corresponding color palette. Our proposed model consists of two conditional generative adversarial networks: the text-to-palette generation networks and the palette-based colorization networks. The former employs a sequence-to-sequence model with an attention module to capture the semantics of the text input and produce relevant color palettes. The latter utilizes a U-Net architecture to colorize a grayscale image using the generated color palette. Our evaluation results show that people preferred our generated palettes over ground truth palettes and that our model can effectively reflect the given palette when colorizing an image.READ FULL TEXT VIEW PDF
Text-to-Image translation has been an active area of research in the rec...
Generating novel pairs of image and text is a problem that combines comp...
The sequence to sequence architecture is widely used in the response
Automatic lyrics generation has received attention from both music and A...
We propose a new task in the area of computational creativity: acrostic ...
This paper addresses the problem of manipulating images using natural
Synthetic data used for scene text detection and recognition tasks have
Implementation of our paper, Text2Colors: Guiding Image Colorization through Text-Driven Palette Generation
Humans can associate certain words with certain colors. The real question is, can machines effectively learn the relationship between color and text? Using text to express colors can allow ample room for creativity, and it would be useful to visualize the colors of a certain semantic concept. For instance, since colors can leave a strong impression on people , corporations often decide upon the season‘s color theme from marketing concepts such as ‘passion.’ Through text input, even people without artistic backgrounds can easily create color palettes that convey high-level concepts. Since our model uses text to visualize aesthetic concepts, its range of future applications can encompass text to even speech.
Previous methods have a limited range of applications as they only take a single word as input and can recommend only a single color or a color palette in pre-existing datasets [12, 8, 15, 25]. Other studies have further attempted to link a single word with a multi-color palette [21, 36] since multi-color palettes are highly expressive in conveying semantics . Compared to these previous studies, our model can generate multiple plausible color palettes when given rich text input, including both single- and multi-word descriptions, greatly increasing the boundary of creative expression through words.
In this paper, we propose a novel method to generate multiple color palettes that convey the semantics of rich text and then colorize a given grayscale image according to the generated color palette. Perception of color is inherently multimodal , meaning that a particular text input can be mapped to multiple possible color palettes. To incorporate such multimodality into our model, our palette generation networks are designed to generate multiple palettes from a single text input. We further apply our generated color palette to the colorization task. Motivated from previous user-guided colorizations that utilize color hints given by users [42, 45], we design our colorization networks to utilize color palettes during the colorization process. Our evaluation demonstrates that the colorized outputs do not only reflect the colors in the palette but also convey the semantics of the text input.
The contribution of this paper includes:
(1) We propose a novel deep neural network architecture that can generate multiple color palettes based on natural-language text input.
(2) Our model is able to use the generated palette to produce plausible colorizations of a grayscale image.
(3) We introduce our manually curated dataset called Palette-and-Text (PAT), which includes 10,183 pairs of a multi-word text and a multi-color palette. 111Dataset and codes are publicly available at https://github.com/awesome-davian/Text2Colors/
Meanings associated with a color are both innate and learned . For instance, red can make us instinctively feel alert . Since color has a strong association with high-level semantic concepts , producing palettes from text input is useful in aiding artists and designers  and allows automatic colorization from palettes [42, 5]. A downside to using text to choose a filter is that filter names do not usually convey the filter’s colors , thus making it difficult for users to find the filter that matches their taste just by looking at filter names. To bridge this discrepancy between color palettes and their names, palette recommendation based on user text input has long been studied. Query-based methods [21, 36] use text inputs to query an image from an image dictionary where colors are extracted from the queried image to make an associated palette. This method is problematic in that the text input is mapped to the image content of the queried image rather than the color that the text implies. Instead of looking for a target directly, learning-based approaches [14, 27, 23] match or generate color palettes to their linguistic descriptions by learning their semantic association from large-scale data. However, our model is the only generative model that supports phrase-level text input.
Conditional generative adversarial networks (cGAN) are GAN models that use conditional information for the discriminator and the generator . cGANs have drawn promising results for image generation from text [32, 31, 43]16, 13, 7]. StackGAN  is the first model to use conditional loss for text to image synthesis. Our model is the first to utilize the conditioning augmentation technique from StackGAN to output diverse palettes even when given the same input text.
Colorization is a multimodal task and desired colorization results for the same object may vary from person to person . A number of studies introduce interactive methods that allow users to control the final colorization output [45, 20]. In these models, users directly interact with the model by pinpointing where to color. Even though these methods achieve satisfactory results, a limitation is that users need to have a certain level of artistic skill. Thus instead of making the user directly color an image, other studies take a more indirect approach by utilizing color palettes to recolor an image [3, 5]. Palette-based filters of our model are an effective way for non-experts to recolor an image .
This section introduces our manually curated dataset named Palette-and-Text (PAT). PAT contains 10,183 text and five-color palette pairs, where the set of five colors in a palette is associated with its corresponding text description as shown in Figs. 3(b)-(d). Words vary with respect to their relationships with colors; some words are direct color words (e.g., pink, blue, etc.) while others evoke a particular set of colors (e.g., autumn or vibrant). To the best of our knowledge, there has been no dataset that matches a multi-word text and its corresponding 5-color palette. This dataset allows us to train our models for predicting semantically consistent color palettes with textual inputs.
, is a well-established multi-color dataset. Kobayashi only uses 180 adjectives to express 1170 three-color palettes, which greatly limits its range of expression. In contrast, our dataset is made up of 4,312 unique words. This includes much more text that was not traditionally used to express colors. Our task requires a more sophisticated dataset like PAT, that matches a text to multiple colors and is large enough for a deep learning model to learn from.
We generated our PAT dataset by refining user-named palette data crawled from a community website called color-hex.com. Thousands of users upload custom-made color palettes on color-hex, and thus our dataset was able to incorporate a wide pool of opinions. We crawled 47,665 palette-text pairs and removed non-alphanumerical and non-English words. Among them, we found that users sometimes assign palette names in an arbitrary manner, missing their semantic consistency with their corresponding color palettes. Some names are a collection of random words (e.g., ‘mehmeh’ and ‘i spilled tea all over my laptop rip’), or are riddled with typos (e.g., ‘cause iiiiii see right through you boyyyyy’ and ‘greene gardn’). Thus, using unrefined raw palette names would hinder model performances significantly.
To refine the noisy raw data, four annotators voted whether the text paired with the color palette properly matches its semantic meanings. We then used only the text-palette pairs in which at least three annotators out of four agreed that semantic matching exists between the text and color palette. Including text-palette pairs in the dataset only when all four annotators agree was found to be unnecessarily strict, leaving not much room for personal subjectivity. Annotator’s perception is inherently subjective, meaning that a text-palette pair perfectly plausible to one person may not be agreeable to another. We wanted to incorporate such subjectivity by allowing a diverse selection of text-palette pairs. Mis-spelling and punctuation errors were manually corrected after the annotators finished sorting out the data.
Text2Colors consists of two networks: Text-to-Palette Generation Networks (TPN) and Palette-based Colorization Networks (PCN). We train the first networks to generate color palettes given a multi-word text and then train the second networks to predict reasonable colorizations given a grayscale image and the generated palettes. We utilize conditional GANs (cGAN) for both networks.
be word vectors initialized by 300-dimensional pre-trained vectors from GloVe. Words not included in the pre-trained set are initialized randomly. Using the CIE Lab space for our task, represents a 15-dimensional color palette consisting of five colors with Lab values. After a GRU encoder encodes into hidden states , we add random noise to the encoded representation of text by sampling latent variables
from a Gaussian distribution. The sequence of conditioning vectors is given as condition for the generator to output a palette , while its mean vector is given as the condition for the discriminator. Our objective function of the first cGAN can be expressed as
where discriminator tries to maximize against generator that tries to minimize . The pre-trained word vectors and the real color palette is sampled from true data distribution .
Previous approaches have benefited from mixing the GAN objective with distance  or distance . We have explored previous loss options and found the Huber (or smooth ) loss to be the most effective in increasing diversity among colors in generated palettes. The Huber loss is given by
This loss term is added to the generator’s objective function to force the generated palette to be close to the ground truth palette. We also adopted the Kullback-Leibler (KL) divergence regularization term , i.e.,
which is added to the generator’s objective function to further enforce the smoothness over the conditioning manifold. Our final objective function is
are the hyperparameters to balance the three terms in Eq.6. We set in our model.
Learning a mapping from text to color is inherently multimodal. For instance, a text ‘autumn’ can be mapped to a variety of plausible color palettes. As text becomes longer, such as ‘midsummer to autumn’ or ‘autumn breeze and falling leaves’, the scope of possible matching palettes becomes more broad and diverse. To appropriately model the multimodality of our problem, we utilize the conditioning augmentation (CA)  technique. Rather than using the fixed sequence of encoded text as input to our generator, we randomly sample latent vector from a Gaussian distribution as shown in Fig. 5. This randomness allows our model to generate multiple plausible palettes given same text input.
To obtain the conditioning variable , the pre-trained word vectors are first fed into a GRU encoder to compute hidden states . This text representation is fed into a fully-connected layer to generate and (the values in the diagonal of ) for the Gaussian distribution . Conditioning variable is computed by , where is the element-wise multiplication and . The resulting set of vectors will be used as condition for our generator.
is a GRU hidden state vector for time , having the previously generated color , the context vector , and the previous hidden state as input. The GRU hidden state is given as input to a fully-connected layer to output the -th color of the palette . The resulting five colors are combined to produce a single palette output .
The context vector depends on a sequence of conditioning vectors and the previous hidden state . The context vector is computed as the weighted sum of these conditions ’s, i.e.,
The weight of each conditional variable is computed by
is a sigmoid activation function andis a weight vector. The additive attention  computes how well the -th word of the text input matches the -th color of the palette output. The score is computed based on the GRU hidden state and the -th condition . The attention mechanism enables the model to effectively map complex text input to the palette output.
For the discriminator , the conditioning variable
and the color palette are concatenated and fed into a series of fully-connected layers. By jointly learning features across the encoded text and palette, the discriminator classifies whether the palettes are real or fake.
The goal of the second networks is to automatically produce colorizations of a grayscale image guided by the color palette as a conditioning variable. The inputs are a grayscale image representing the lightness in CIE Lab space and a color palette consisting of five colors in Lab values. The output corresponds to the predicted ab color channels of the image. The objective function of the second model can be expressed as
and included in the equation are shown in Fig.4. We have also added the Huber loss to the generator’s objective function. In other words, the generator learns to be close to the ground truth image with plausible colorizations, while incorporating palette colors to the output image to fool the discriminator. We set in our model.
The generator consists of two sub-networks: the main colorization networks and the conditioning networks. Our main colorization networks adopts the U-Net architecture , which has shown promising results in colorization tasks [13, 45]. The skip connections help recover spatial information , as the input and the output images share the location of prominent edges .
The role of the conditioning networks is to apply the palette colors to the generated image. During training, the networks are given a palette extracted from the ground truth image . We utilize the Color Thief 222http://lokeshdhakar.com/projects/color-thief/ function to extract a palette consisting of five dominant colors of the ground truth image. Similar to the previous work , the conditioning palette is fed into a series of conv-relu
conv-relulayers as shown in Fig. 4. The feature maps in layers 1, 2, and 4 are duplicated spatially to match the spatial dimension of the , , and features in the main colorization networks and added in an element-wise manner. The palette is fed into upsampling layers with skip connections as well as the middle of the main networks. This allows the generator to detect prominent edges and apply palette colors to suitable locations of the image. During test time, we use the generated palette from the first networks (TPN) as the conditioning variable, colorizing the grayscale image with the predicted palette colors.
As our discriminator , we use a variant of the DCGAN architecture . The image and conditioning variable are concatenated and fed into a series of conv-leaky relu layers to jointly learn features across the image and the palette. Afterwards, it is fed into a fully-connected layer to classify whether the image is real or fake.
We first train and
of TPN for 500 epochs using the PAT dataset. We then trainand of the PCN for 100 epochs, using the extracted palette from a ground truth image. Finally, we use the trained generators and during test time to colorize a grayscale image with generated palette from a text input . All networks are trained using Adam optimizer 
with a learning rate of 0.0002. Weights were initialized from a Gaussian distribution with zero mean and standard deviation of 0.05. We set other hyper parameters as, and .
This section presents both quantitative and qualitative analyses of our proposed model. We evaluate the TPN (Section 4.1) based on our PAT dataset. For the training of the PCN (Section 4.2), we use two different datasets, CUB-200-2011 (CUB) 
and ImageNet ILSVRC Object Detection (ImageNet dataset).
This section discusses the evaluation on multimodality and diversity of our generated palettes. Multimodality refers to how many different color palettes a single text input can be mapped to. In other words, if a single text can be expressed with more color palettes, the more multimodal it is. As shown in Fig. 6, our model is multimodal, while previous approaches are deterministic, meaning that it generates only a particular color palette when given a text input. Diversity within a palette refers to how diverse the colors included in a single palette are. Following the current standard for perceptual color distance measurement, we use the CIEDE2000  on CIE Lab space to compute a model’s multimodality and diversity. To measure multimodality, we compute the average minimum distances between colors from different palettes. To measure diversity of a color palette, we measure the average pairwise distance between the five colors within a palette. All measurements are computed based on the test dataset.
Table 1 shows the multimodality and diversity measurement among the variants of our model. The CA module (Section 4.1.2) enables our networks to suggest multiple color palettes when given the same text input. The model variant without CA (the first row in Table 1) results in zero multimodality, indicating that the networks generate identical palettes for the same text input. Another palette generation model by Heer and Stone  also has zero multimodality. This shows that TPN is the only existing model that can adequately express multimodality, which is crucial in the domain of colors. Although Heer and Stone’s model has higher diversity than TPN, Fig. 6 shows that their palettes contain irrelevant colors that may increase diversity but decrease palette quality. On the other hand, TPN creates those palettes containing colors that well match each other. Results on the fooling rate will be further illustrated in Section 5.3.
The attention module (Section 4.1.2) plays a role of attending to particular words in text input to predict the most suitable colors for the text input. Fig. 7 illustrates how the predicted colors are influenced by attention scores. The green-colored boxes show attention scores computed for each word token when predicting each corresponding color in the palette. Higher scores are indicated by dashed-line boxes. We observe that three colors generated by attending to ghoul are all dark and gloomy, while the other two colors attending to fun are bright. This attention mechanism enables our model to thoroughly reflect the semantics included in text inputs of varying lengths.
|Palette Evaluation||User Study: Part I|
|Diversity||Multimodality||Fooling Rate (%)|
|Heer and Stone||-|
Ground truth palette
We conduct a user study to reflect universal user opinions on the outputs of our model. Our user study is composed of two parts. The first part measures how the generated palettes match the text inputs. The second part is a survey that compares the performance of our palette-based colorization model to another state-of-the-art colorization model. 53 participants took part in our study.
Our goal is to generate a palette with a strong semantic connection with the given text input. A natural way to evaluate it is to quantify the degree of connection between the text input and the generated palette, in comparison to the same text input and its ground truth palette. Given a text input, its generated palette, and the ground truth palette, we ask human observers to select the palette that best suits the text input. A fooling rate (FR) in this study indicates the relative number of generated palettes chosen over ground truth palettes. More people choosing the generated palette results in a higher FR. This measure has often been used to assess the quality of colorization results [45, 11]. We will use this metric to measure how much a text input matches its generated palette.
Users participate in the user study over TPN and Heer and Stone’s model . Each consists of 30 evaluations. We randomly choose a single data item out of 992 test data and show the text input along with the generated palette and the ground truth palette.
In Table 1, we measure the FR score for each person and compute the mean and the standard deviation (std) of all of the scores from participants. Max and min scores represent the highest and the lowest FR scores, respectively, recorded by a single person. While Heer and Stone’s model  shows low FR of 39.6%, our TPN has the FR of 56.2% while maintaining a high level of diversity and multimodality. The FR of 56.2% indicates that the generated palettes are indistinguishable to human eyes and sometimes even match the input text better than the ground truth palettes. Note that the standard deviation of 12.7% implies diverse responses to the same data pairs.
In this part of the user study, we conduct a survey on the performance of the PCN given palette inputs. Users are asked to answer five questions based on the given grayscale image, the color palette, and the colored image. For quantitative comparison, we set a state-of-the-art colorization model  as our baseline. This model originally contains local and global hint networks. In our implementation of the baseline model, we utilize the global hint networks to infuse our generated palette to the main colorization networks. Note that we modified the baseline model to fit our task. Our novelty is the ability to produce high-quality colorization with only five colors of a palette while our baseline  needs 313 bins of ab gamut. Our model is able to colorize with limited information due to novel components such as the conditional adversarial loss and feeding the palette into skip-connection layers.
We show colorization results of our PCN and the baseline model one-by-one in a random order. Then, we ask each participant to answer five different questions (shown in Fig. 9) based on a five-point Likert scale. The focus of our questions is to evaluate how well the palette was used in colorizing the given grayscale image. The total number of data samples per test is 15.
The resulting statistics are reported in Fig. 9. Our PCN achieves higher scores than the baseline model across all the questions. We can infer that the palettes generated by our model are preferred over palettes created by a human hand. Since our model learns consistent patterns from a large number of human-generated palette-text pairs, our model may have generated color palettes that more users could relate to.
We proposed a generative model that can produce multiple palettes from rich text input and colorize grayscale images using the generated palettes. Evaluation results confirm that our TPN can generate plausible color palettes from text input and can incorporate the multimodal nature of colors. Qualitative results on our PCN also show that the diverse colors in a palette are effectively reflected in the colorization results. Future work includes extending our model to a broader range of tasks requiring color recommendation and conducting the detailed analysis of our dataset.
Acknowledgement. This work was partially supported by the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIP) (No. NRF2016R1C1B2015924). Jaegul Choo is the corresponding author.
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: Proc. the International Conference on Learning Representations (ICLR) (2014)
Charpiat, G., Hofmann, M., Schölkopf, B.: Automatic image colorization via multimodal predictions. In: Proc. the European Conference on Computer Vision (ECCV) (2008)
Cho, J., Yun, S., Lee, K., Choi, J.Y.: PaletteNet: Image recolorization with given color palette. In: Proc. the IEEE Conference on Computer Vision and Pattern Recognition Workshops (2017)
Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: Conference on Empirical Methods in Natural Language Processing (EMNLP) (2014)
Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proc. the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
Kim, T., Cha, M., Kim, H., Lee, J., Kim, J.: Learning to discover cross-domain relations with generative adversarial networks. In: Proc. the International Conference on Machine Learning (ICML) (2017)
Our user study consists of two parts, one for evaluation of Text-to-palette Generation Networks (TPN) and the other for evaluation of Palette-based Colorization Networks (PCN). Fig. 11(a)-(b) illustrates how our data tuples were shown to the participants in Part I and Part II, respectively.
Fig. 12 shows comparisons of color distributions between ground truth palettes of the training data and generated palettes from our test data. For each color distribution, we quantize the ab values of every palette color into 313 color bins 
and visualize the probability distribution ofab values. We compare three model variants of different objective functions: cGAN+Huber (=100), Huber (=100), and cGAN (=0). We also compute the Kullback-Leibler (KL) divergence between the ground truth palette distribution of the training data and that of our model variants.
As shown in the bottommost plot of Fig. 12, the Huber loss plays a critical role in producing proper colors close to the ground truth image. Without the Huber loss, the model does not only fail to recover the color distribution similar to the ground truth data but also exhibits the lowest fooling rate of 30.7% in user study results. On the other hand, the model with cGAN+Huber loss (=100) records the lowest KL divergence of 0.2299 as well as the best fooling rate of 56.2%, while the model with only the Huber loss (=100) records the second best. This is due to the fact that only using the Huber loss leads to blindly averaging over multiple ground truth palettes, resulting in slightly desaturated palette results as shown in the second row of Fig. 13. In contrast, the model with both cGAN+Huber loss learns and preserves various ground truth colors rather than simply averaging them, resulting in bright, highly saturated results as shown in the first row of Fig. 13.
This section shows additional, diverse and detailed results from TPN.
Fig. 14 shows how our model handles phrase-level inputs. To make comparison easier, all the phrases contains the word ‘love.’ It is interesting to see how our model chooses to express the subtle nuance differences included in the input text. Notice how the output color palettes tend to be darker for text inputs that are negative towards ‘love’ (e.g., ‘i thought i loved you’ and ‘where did our love go’). All input phrases included in this figure are unseen data.
Fig. 15 shows outputs of our model in comparison to ground truth palettes. If an input word is seen at least once in the training data, our model is able to output a color palette related to the input word. For instance, take a look at the color palette named ‘mango and grapefruit’ on the top left. The word ‘grapefruit’ is included only once in the training set. Yet, the model successfully outputs a color palette that matches the text input. Also, ground truth palettes are included for a direct comparison with generated palettes. Even if the predicted palette is not exactly identical to the ground truth palette, both can be perceived as reasonable colors.
Even though our model can effectively produce semantically meaningful colorizations, it struggles when unknown tokens are given as input. Unknown tokens refer to words not included in the training set. it is not surprising that our model fails and outputs the same washed-out grayish-brown palettes as we can see in Fig. 16. On the other hand, our model can still produce reasonable palettes in the case of unseen, new combinations of words found in the training set. For example, ‘bright life’ in Fig. 15 was seen separately as ‘bright’ and ‘life’ in the training set but not together. Thus, ‘bright life’ is classified as unseen data, which our model has no problem in predicting color palettes from.
We present additional colorization results on datasets including CUB-200-2011 (CUB dataset) , ImageNet ILSVRC Object Detection (ImageNet dataset) , and Graphical Pattern images (Pattern images) in Figs. 17-22. In these figures, the leftmost columns are grayscale images. Text inputs are given above the grayscale image. The vertical color palettes next to the grayscale images are palettes generated from the text input. The output has been colorized with the generated color palette. We would like to emphasize that our model effectively utilizes the generated color palettes during the colorization process. The colorized image may be different from its natural colors because our networks incorporate additional color hints. We display the original ground truth image on the right to compare how different an image becomes after applying the palettes.
Our PCN model generalizes surprisingly well on other types of images. Our model is trained on ImageNet dataset, which is mostly made up of natural images. Instead of natural images, we used our colorization model to colorize graphical pattern images. The graphical pattern images are crawled from Google through searching keywords such as ‘pattern,’ ‘fabric pattern,’ or ‘beautiful patterns.’ As seen in Figs. 21 and 22, graphical pattern images are significantly different from natural images. The colorized outputs show that our model can apply our generated color palettes on images of diverse shapes and textures. The results qualitatively show that our palette-based colorization model is transferable to other image domains.