Evolutionary search has frequently been used to generate artistic images [4, 3, 14]. Images have a high-dimension. Previous work has either reduced the dimensionality of the search space through programmatic encodings [3, 18] or have constrained the images with priors [21, 1]. In recent years, Generative Adversarial Networks GAN
’s have been used to map a low dimensional real-valued latent vector into images in the category for which theGAN has trained. There has been work in using GAN’s to generate and mix novel images  and perform artistic style transfer , along with many other applications. However, to date there has no work using evolutionary, or other methods, to explore the latent space of a GAN to generate images according to aesthetic feature measures. In this work we employ GAN’s to generate novel images by evolving the latent vector to maximise and minimise single aesthetic and pairs of aesthetic features. We show that the generation of images in this space requires the use of carefully constructed constraints on image realism. We also show that different GAN’s appear to impose different bounds on the values of aesthetic measures that can be evolved.
The paper is structured as follow. Section 1 outlines related work. Section 2 presents the methodology used for evolving images. Section 3 and 4 present our results. Finally, section 5 presents discussion and the future work.
Aesthetic feature measures have been often applied to the creation of new artistic images using evolutionary search [4, 3, 14, 16]. There has also been significant work in the evolution of existing images [22, 20].
This work differs from previous work in the use of a GAN as a mapper from the latent search vector to the images space and the use of the discriminator network and feature metrics to constrain these images, the space of a generator network as a given image source and use the discriminator and feature metrics to constrain these images. In terms of deep learning, Gatys [10, 11, 9] used a convolutional network to transfer artistic style into an existing image. These new approaches in network architectures and training methods enabled the generation of realistic images [24, 7]. Recently Dosovitskiy and Brox  trained networks of generating images from feature vector and combining an auto-encoder-style approach with deep convolutional generative adversarial networks training. Furthermore, Nguyen 
used priors from Deep Generative Network to generate image variants that look close to natural within a preferred inputs for neurons.
for Traveling Salesman Problem (TSP) which is a NP-hard combinatorial optimization problem with real world applications.
In this section we describe the methodology that we use to evolve images. In our system, images are created by optimising the latent space of the GAN to create images that score high or low on aesthetic feature measures. Further, we try to constrain images to be real, this is done using the discriminator network trained with the GAN.
In this section we describe our system that is based on Generative Adversarial Networks GAN . Figure 1 shows the structure of our system. In the GAN, we train two networks: 1) a generator, to generate images from a latent vector of 100 real numbers, and 2) a discriminator which scores the images in the generator for realness as both networks are trained. We train these two networks with two image datasets: the Celebfaces attributes (CelebA) data set containing more than 150k celebrity faces 
and the imagenet class of butterflies containing over butterfly images.
is based on the PytorchGAN tutorial . The generator component of the network consists of
deconvolutional layers. The activation functions for the first
layers are LeakyRelu’s, and the hidden layer is a ReLU, and the last deconvolutional layer uses tanh. The discriminator uses LeakyReLU’s in its convolutional layers. The generator takes the hundred elements of
as input and generates a 128x128 pixel image. The discriminator takes an image and generates a normally distributedrealness score with the most real images scoring zero. Finally, at the top of Figure 1 is the aesthetic feature function, as described in .
The GAN and the necessary feature functions are linked together to drive evolution. The combined system works as follows. A randomly initialised latent feature vector is sent into the generator, which outputs an image. This image is run through both the chosen artistic feature function and the discriminator, and both contribute to a score. The evolutionary process, guided by the score, mutates with the goal of optimising both the realness and the desired artistic feature of the output image.
This section describes in more detail the features used in our experiments. We denote a function for an image , representing an artistic feature. This function maps an image to a scalar value . For our experiments we use the following features: mean hue, saturation, smoothness, reflectional symmetry and Global Contrast Factor. We describe the features as follows.
Mean-hue is the mean value of every pixel’s hue in the image. The range of Hue is . Note that both and represent the colour red.
Mean-saturation is the mean value of every pixel’s saturation in the image. The range is with representing low Saturation and representing high.
Smoothness of an image is measured, for a given picture with pixels as:
where gradient is the gradient magnitude image produced by MATLAB’s intermediate image gradient method, which calculated gradients between adjoining pixel values on each colour channel. We perceive from this that is the disparity of colour between adjacent pixels and also lies within the range .
Reflectional Symmetry is a measure based on den Heijer’s work  to measure the degree which an image reflects itself. Symmetry divides an image into four quadrants and measures horizontal, vertical, and diagonal symmetry. Note Symmetry is defined for image as:
Global Contrast Factor is a measure of mean contrast between neighbouring pixels at different image resolutions. GCF is determined by calculating the local contrast at each pixel at resolution :
where is the perceptual luminosity of pixel and are the four neighbouring pixels of at resolution . The mean local contrast at the current resolution is defined as:
From these local contrasts, GCF is calculated as
The pixel resolutions correspond to different superpixel sizes of , and . Each superpixel is set to the average luminosity of the pixel’s it contains. The are empirically derived weights of resolutions from  giving highest weight to moderate resolutions. Note GCF’s range is not bounded to .
In this work we investigated the use of single and multi-feature optimization. We explore optimization space with respect to feature values. For a single feature our system minimise and maximise particularly feature. We define the minimization process feature and for maximization process feature111Note that for featureGCF we are able to maximize and scale is in the range .. For double features we have four optimisaton targets representing the combinations of minimising and maximising and .
To maintain realness we penalise the combined asthetic score with a measure for realness from the discriminator . Thus our fitness functions for single features are shown in equations and for multi-features in .
For multi-feature experiments we use feature combinations: hue-saturation, hue-symmetry, saturation-symmetry, smoothness-saturation, GCF-smoothness and GCF-saturation. These combinations were chosen to produce potentially interesting outputs. GCF + smoothness and GCF + saturation were specifically chosen due to related work indicating GCF + smoothness would constrain each other , resulting in lower image diversity.
3 Preliminary Experiments
To ensure meaningful results, we refined the methodology through an experimental process. These refinements are discussed in following.
Initial experimentation was done using (1+1) EA and CMA-ES  frameworks to determine the performance of both algorithms. CMA-ES was allowed to run for mutations (the equivalent of iterations) and (1+1) EA was allowed to run for iterations to promote a valid comparison.
As illustrated in Figure 2 CMA-ES was able to achieve more extreme feature values. This superiority of optimisation applied to all feature metrics. The CMA-ES ran for 80 minutes and (1+1) EA taking 240 minutes for an optimisation run.
Feature Experiments without Realness Constraint
It was initially assumed that the GAN would be able to create face-like images with any input vector. Some tests were performed which did not incorporate a constraint on realness as part of the optimisation process. As Figure 4 demonstrates optimising features without the discriminator produces abstract images.
To constrain images to be more realistic three constraining methods were tested: 1) limiting the degree CMA-ES was allowed to mutate ; 2) discarding images that failed a certain realness threshold and 3) incorporating the discriminator’s return value into the optimisation function. It was found that the third option of integrating realness as a variable gave the best results. The option of discarding images resulted in CMA-ES failing progress. Restricting the values in to near-zero do not preserve realism as much as using the discriminator.
Single Dimensional Feature Experiments with Constraint
Single feature experiments require the fewest variables to optimise and as such, could be expected to evolve the image with the least difficulty. The results of each feature value are shown in Table 1.
As can be seen the ranges of features above are very small, with the exception of symmetry. In these runs the term has the strong effect of constraining the feature values. For symmetry, the larger range might be explained by presence of both symmetric and asymmetric faces in the training dataset. In line with the small feature ranges the constrained images only showed small differences as seen in figure 3 corresponding to the hue and symmetry measures in Table 1.
Two-Dimensional Feature Experiments with Constraint
Running the experiments on multiple features gave similarly constrained images to the single features. Figure 5 shows images evolved to minimise and maximise saturation and symmetry.
As can be seen in the Figure 5, there is some success in evolving different amounts of symmetry but not a particularly strong difference in saturation. It appears, at least for some features, the realness constraint is preventing strong exploration of the feature space.
Impact of cut-off function
In order to maintain balance between image realness and exploration we modify the discriminator term by passing the raw result of for an image through a cut-off function define as follow:
with a cutoff and stable value . In the experiments that follow we set , returning maximum realness, and vary to test its effect.
With the cut-off function, the search is unaffected until the image reaches a certain threshold of realness. When the threshold is reached the system has no variation in respect to the realness value, thus giving priority to aesthetic features over realness values.
A subjective analysis of possible cut-off values needed to be performed in order to determine the optimal value for future experiments. Figure 6 demonstrates the effect of different cut-off values on both image realness and aesthetic feature value.
A relationship can be observed from the above images. As the cut-off decreases, so does the degree we are able to evolve the feature value. However, it can be noted that even the cut-off was able to create a far lower hue compared to the constrained results – while still being realistic enough to be called a face. The cut-off was used in the remaining single feature experiments.
This section presents the results of evolution in single and paired feature spaces. The previous experiments were all carried on faces. In the following GAN’s generated from both the faces and the butterfly datasets were used.
Single Dimensional Feature Experiments with Cut-off Function
We conducted single feature dimension experiments using the face and butterfly datasets for the following features: hue, saturation, smoothness, symmetry and GCF. For these experiments we use a cut-off of on the discriminator output for both GAN’s. The results were obtained for the minimum and maximum feature values from each experiment. Figure 7 shows the result of the experiments for the single dimensional feature with each row corresponding to an image minimising and maximising the feature for faces and butterflies respectively.
Table 4 shows the minimum and maximum value for each feature for the faces and butterfly datasets, respectively. We observe that hue has the highest range with respect to feature values. The use of the butterfly dataset provides good way to see how evolution with aesthetic measure responds to the priors embedded in different GAN’s.
For the single dimensional experiments we observe in Figure 7 that images generated with the faces dataset appear more real than the images generated with butterfly dataset. This is likely to be due to the more diverse nature of the butterfly dataset. The images shown in Figure 7
(a) have the most variance in the hue dimension. Image with lowest hue value appears most realistic in contrast the image with higher value appears less realistic. We observe that image generated from the butterfly dataset achieves a higher feature range.
Figure 7 (b) shows that in spite of the saturation feature for faces extending over a narrow range the resulting faces are not very realistic. The butterfly dataset is able to produce higher values of saturation, resulting in a realistic and colorful image. In Figure 7 (c) we observe that minimization for produces realistic images with superimposed darker shadows. In contrast maximising smoothness produces less realistic images.
The images shown in Figure 7 (d) produces high values for symmetry for both datasets. These images appears symmetrical and less real. Images with lower symmetry value are more real.
Finally, the images shown in Figure 7 (e) exhibits less realness for faces and butterflies dataset in minimisation and more realism in maximisation.
Two-Dimensional Feature Experiments with Cut-off Function
In our next experiment, we evolve images using the GAN to minimise and maximise in two feature dimensions. These experiments aim to give us insight into how features interact with each other and also the impact of the image priors as embedded in the GAN’s on the extent to which features can be optimised. After training our GAN models on the faces and butterfly dataset, We run experiments with the following feature combinations: GCF-Saturation; GCF-Smoothness; Hue-Saturation; Hue-Symmetry; Saturation-Symmetry; and Smoothness-Saturation.
The feature, pair values resulting from these experiments are shown in Tables 4 (for faces) and 4 (for butterflies). The images corresponding to these values are shown in Figure 8 and 9. The first column of Figures8 and 9, show images, in clockwise order from top-left, for min-max, max-max, max-min, min-min combinations of features for faces, The corresponding pictures butterflies dataset is shown in the second column. The last column, plots the positions in feature-space, of the four face images from the first column (in red) and the four butterfly images from the second column (in blue). The shape of the quadrilateral in these plots provides an indication of how feature values are constrained with respect to each other and by the GAN used to generate them.
Based on our findings from the previous experiments with single dimension diversity, we reduced the cut-off to for the multi-feature experiments to try to maintain the realism of the images. The impact of this smaller cut-off can be observed in Figures 8 and 9 in terms of the relatively small areas of feature space contained by the plots.
Looking at the feature-pairs in turn. The images shown in Figure 8 (a) have the highest GCF values and for maxmin and maxmax optimisation appear most realistic. We observe in Figure 8 (b) that image generated on the butterflies dataset achieve higher score for GCF and permit a higher range of saturation for high-GCF.
The feature plot in Figure 8 shows that GCF and Saturation can vary independently. In contrast, Figure 8 (b)’s plot indicates some difficulty in minimising both smoothness and GCF. From the plot also appears to be relatively difficult to maximise both of these features. This result is in concordance with the observations in  which found that GCF and smoothness, being spatial features, appeared to be in conflict with each other. More specifically the high contrast required for high GCF scores is in direct conflict with the low contrast required for high smoothness scores. Also notable from figure 8 (b) is a relative lack of realism in the faces as compared to those in part (a).
The images shown in Figure 9 (a) show the relationship between hue and saturation. The butterfly pictures show the most variance in the saturation dimension and the face pictures show marginally more variance in hue. Images high in saturation seem to appear sharper – with the face image that maximises both features having quite harsh colour, more contrast, and a mask-like appearance.
For figure 9 (b) both sets of images have similar ranges of symmetry but faces have a much narrower range of hue. Highly symmetric images seem to be less realistic, tending to ovoid shapes, with detail seemly sacrificed in order to maximise symmetry. In contrast asymmetric images appear to have more realistic textures and more intense colours.
Figure 9 (c) combines saturation and symmetry. As before highly symmetric images appear less realistic. The evolutionary process seems to have difficulty maximising both saturation and symmetry for both GAN’s. Clearly it is possible to create artificial images that score highly on both feature dimensions so this difficulty may be reflective of the rarity of this feature combination in the training sets for these GAN’s. As a final observation, the butterfly picture maximising both features resembles an insect’s face, perhaps an interesting consequence of having diverse images in the training set.
Finally, the images in Figure 9 (d) show difficulty in minimising both smoothness and saturation. There is a smaller corresponding problem in maximising both features. In both data sets the most realistic images are produced by the minimisation of smoothness and the maximisation of saturation, perhaps indicating that the priors in the data set are biased toward rougher and more colourful images.
|0.0426 - 0.3318||0.0457 - 0.3671||0.2900 - 0.2727||0.4651 - 0.4257|
|0.0480 - 0.7233||0.0549 - 0.9577||0.2523 - 0.6448||0.0935 - 0.9722|
|0.2573 - 0.7020||0.3190 - 0.9654||0.4256 - 0.6336||0.3582 - 0.9679|
|0.9814 - 0.2378||0.9751 - 0.4876||0.9901 - 0.2912||0.9903 - 0.3670|
|0.0164 - 0.2930||0.0126 - 0.4048||0.0310 - 0.2796||0.0332 - 0.4160|
|0.0166 - 0.9890||0.0166 - 0.9890||0.0326 - 0.9762||0.0290 - 0.9893|
|0.1622 - 0.1956||0.1218 - 0.4772||0.4659 - 0.1402||0.3458 - 0.5912|
|0.1286 - 0.7232||0.1744 - 0.9558||0.5067 - 0.6701||0.3075 - 0.9614|
|0.1447 - 0.7133||0.1594 - 0.9554||0.5760 - 0.6512||0.3014 - 0.9647|
|0.9840 - 0.1469||0.9706 - 0.6177||0.9852 - 0.1291||0.9868 - 0.4661|
|0.0100 - 0.1743||0.0102 - 0.3820||0.0334 - 0.1983||0.0357 - 0.5205|
|0.0094 - 0.9826||0.0093 - 0.9840||0.0378 - 0.9638||0.0319 - 0.9838|
5 Discussion and future work
Evolutionary search can be a powerful technique for creating novel images. In this work, we have shown how to apply GAN’s in order to generate images scoring high or low for given aesthetic feature values. We used evolutionary search to maximise and minimise single features and pairs of features for two datasets, faces and butterflies.
We have shown how to explore the latent space of a GAN to create semi-realistic images that sample different regions of aesthetic feature spaces. Additionally, we studied the effects of different values of the cut-off function on the aesthetic appearance of the images.
For future research, it would be interesting to explore intermediate points in the feature space to gain more insight into the relationships between features and to explore additional constraints and their effect on the process of generating novel images.
B. Alexander, J. Kortman, and A. Neumann.
Evolution of artistic image variants through feature based diversity
Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2017, Berlin, Germany, July 15-19, 2017, pages 171–178, 2017.
-  Y. Choi. Pytorch Tutorial: Deep Convolutional GAN, https://github.com/yunjey/pytorch-tutorial/tree/master/tutorials/03-advanced/deep_convolutional_gan/.
-  J. Correia, P. Machado, J. Romero, and A. Carballal. Evolving figurative images using expression-based evolutionary art. In Proceedings of the fourth international conference on computational creativity, page 24, 2013.
-  E. den Heijer and A. E. Eiben. Investigating aesthetic measures for unsupervised evolutionary art. Swarm and Evolutionary Computation, 16:52–68, 2014.
-  J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 248–255. IEEE, 2009.
-  A. Dosovitskiy and T. Brox. Generating images with perceptual similarity metrics based on deep networks. In Advances in Neural Information Processing Systems, pages 658–666, 2016.
A. Dosovitskiy, J. T. Springenberg, and T. Brox.
Learning to generate chairs with convolutional neural networks.In Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on, pages 1538–1546. IEEE, 2015.
-  W. Gao, S. Nallaperuma, and F. Neumann. Feature-based diversity optimization for problem instance classification. In International Conference on Parallel Problem Solving from Nature, pages 869–879. Springer, 2016.
-  L. A. Gatys, A. S. Ecker, and M. Bethge. Texture synthesis using convolutional neural networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 262–270, 2015.
-  L. A. Gatys, A. S. Ecker, and M. Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2414–2423, 2016.
-  L. A. Gatys, A. S. Ecker, M. Bethge, A. Hertzmann, and E. Shechtman. Controlling perceptual factors in neural style transfer. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 3730–3738. IEEE Computer Society, 2017.
-  I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672–2680, 2014.
-  N. Hansen, S. D. Müller, and P. Koumoutsakos. Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (cma-es). Evolutionary computation, 11(1):1–18, 2003.
-  T. Kowaliw, A. Dorin, and J. McCormack. Promoting creative design in interactive evolutionary computation. IEEE Trans. Evolutionary Computation, 16(4):523–536, 2012.
-  Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), 2015.
-  P. Machado, J. Romero, and B. Manaris. Experiments in computational aesthetics. In The art of artificial evolution, pages 381–415. Springer, 2008.
-  K. Matkovic, L. Neumann, A. Neumann, T. Psik, and W. Purgathofer. Global contrast factor-a new approach to image contrast. Computational Aesthetics, 2005:159–168, 2005.
-  J. McCormack and M. d’Inverno, editors. Computers and Creativity. Computers and Creativity. Springer, Berlin; Heidelberg, 2012.
V. Nair and G. E. Hinton.
Rectified linear units improve restricted boltzmann machines.
Proceedings of the 27th international conference on machine learning (ICML-10), pages 807–814, 2010.
-  A. Neumann, B. Alexander, and F. Neumann. The evolutionary process of image transition in conjunction with box and strip mutation. In A. Hirose, S. Ozawa, K. Doya, K. Ikeda, M. Lee, and D. Liu, editors, Neural Information Processing - 23rd International Conference, ICONIP 2016, Kyoto, Japan, October 16-21, 2016, Proceedings, Part III, volume 9949 of Lecture Notes in Computer Science, pages 261–268, 2016.
-  A. Neumann, B. Alexander, and F. Neumann. Evolutionary image transition using random walks. In International Conference on Evolutionary and Biologically Inspired Music and Art, pages 230–245. Springer, 2017.
-  A. Neumann, Z. L. Szpak, W. Chojnacki, and F. Neumann. Evolutionary image composition using feature covariance matrices. In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2017, Berlin, Germany, July 15-19, 2017, pages 817–824, 2017.
-  A. Nguyen, A. Dosovitskiy, J. Yosinski, T. Brox, and J. Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 3387–3395. Curran Associates, Inc., 2016.
-  A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.