The ever-growing achievements of complex models relying on powerful learning algorithms, such as those utilized in Deep Learning, have lately started to go beyond their academic boundaries towards their massive deployment in a manifold of application scenarios. This advent has been particularly notable in agriculture , Transportation , Industry 4.0 , and a myriad of mobile applications to help users with their daily lives [4, 5, 6]. However, the design of these modeling techniques is often driven by its performance (let the model perform to its best for the task at hand), thereby overlooking the context in which the model, once trained, will be in use (correspondingly, let the model be usable and understandable by its user). To overcome this issue, quantifying and communicating the possibilities, limitations and caveats of Deep Learning models should be made compulsory for its practicality in real application environments. Unfortunately, the user receiving information on the model might not be an expert on Artificial Intelligence, which ultimately gives rise to a lack of trustworthiness, and his/her eventual reluctance to adopt the designed model in complex application niches.
In this context, current approaches for quantifying the performance of Deep Learning models clashes with the notions an overall user could appraise. For this reason, strategies and methods aimed to bridge the gap between these two realms of knowledge have been profusely explored in recent times. Among them, there are studies that inspect the robustness of Deep Learning models by means of adversarial attacks [7, 8]
. However, the output of these studies is still far from being user-friendly, since their practical implications when done for certain applications cannot be inferred. The estimation of the output confidence of the model has been also addressed in several works[9, 10, 11]. Once again, the result may not be understandable for users without a background in Statistics. Further methods for explaining Deep learning models include the visualization of the internal representation of the model [12, 13], the measurement of the amount of attention placed by the model to each of the inputs  for a certain test sample, and many more covered in recently contributed prospective overviews .
The work presented in this manuscript joins this vibrant research area aimed at making Deep Learning models more interpretable and ultimately, more usable in practice. To this end, we propose an adversarial strategy to produce clear counterfactual explanations of the limits of a Deep Learning classifier. However, we further impose that the generated adversarial samples are plausible
, i.e., changes made on the input to the model have an appearance of credibility and believability without any computer intervention. To ensure plausibility, the proposed method makes use of GANs (Generative Adversarial Networks) in order to learn the underlying probability distribution of each of the features needed to create examples of an objective distribution (e.g. realistic human faces, which serves as an exemplifying use case throughout the paper). In parallel, a Deep Learning convolutional network separates two unknown classes (probability distributions) that are pertaining to the prior distribution (correspondingly, male and female faces). Our method allows for a search among samples of the first distribution to find realistic counterfactuals close to a given test sample that could be misclassified by the model (namely, a face of a male being classified as a female). As a result, our framework makes the user of the model assess its limits with an adversarial analysis of the probability distribution learned by the model, yet maintaining a sufficient level of plausibility for the analysis to be understood by a non-expert user.
The rest of the paper is organized as follows: Section 2 brings an analysis of the background literature for the techniques used in our study, whereas Section 3 presents materials and methods embedded in the proposed framework. Section 4 presents and discusses experimental results and, finally, Section 5 ends the work with several concluding remarks and a prospect of future research lines rooted on our findings.
As anticipated in the introduction, the proposed framework resorts to GANs for producing a realistic counterfactual analysis of image classification models that rely on CNNs (Convolutional Neural Networks). Since the ultimate goal is to favor the understanding of the model classification boundaries by an average user, the framework falls within the XAI (Explainable Artificial Intelligence) umbrella. Consequently, in this section we briefly contextualize and revisit the state of the art of such research areas.
Generative adversarial networks were first introduced by Goodfellow in , bringing the possibility of using neural networks (function approximators) to become generators of a desired distribution. Since their inception, GANs have progressively achieved photo-realistic levels of resolution and quality when synthesizing images of human faces. In general, a GAN architecture consists of two data-based models, which are trained in a mini-max game: one of the players (models) minimizes its error (loss), whereas the other maximizes its gain. In such a setup, multiple models have flourished to date, each governed by its strengths and vulnerabilities . Interestingly for the scope of this paper, some of these were conceived with the intention of finding the pitfalls of a certain model and the ways to hack it [7, 8]. Other GAN approaches aim at generating samples of incredibly complex distributions like photo-realistic human faces [17, 18]. The framework proposed here hybridizes these two approaches by extending what was introduced by , to include the idea of performing an adversarial analysis of a third model.
As for the modeling choice, we adopt the powerful capabilities of CNN [20, 21] to capture spatial correlations from image data. Convolutional layers upon which CNNs are built allow for a space-wise abstraction of the input that, when trained via gradient back-propagation methods, give rise to image classification models of the highest performance. Although CNNs have been under the spotlight for a long time, the availability of huge loads of data and the capability of processing them efficiently has given rise to stacked convolutional architectures that perform incredibly well, at the cost of more complex, less understandable model structures . The more complex the model is, the harder is to pinpoint the reasons for its failures, which motivates studies as the one presented in what follows.
The third technological pillar on which this contribution resides is the recent trend around model explainability, collectively referred to as XAI. In fact, XAI has recently become mainstream in the realm of Artificial Intelligence, unchaining a flurry of different approaches. The recent survey in  provides an extensive review of the current state of the art on this matter, from which we extract [23, 14] as the ones most closely related to the framework proposed in this paper. First, the LIME tool introduced in  allows for a linearization of a certain model’s internal activations when faced with a specific test sample. This attempts to close the gap between the complexity of a model to predict the output and what users can understand to assess their inner functioning. Under this same scope, LRP (Layer-wise Relevance Propagation) implemented in SHAP  helps a user discern where, in the input image, the model focuses towards making its prediction. Once again, the target is to show, in understandable terms (an image overlaid by a heatmap) the reasons why the model outputs its decision. More recently, in  a similar GAN architecture to the one later explained is used to generate images by optimizing for the failure of the discriminator when freezing a couple of target variables. This allows them to study the representativeness of those classes within the dataset that has fed the model. However, a core difference with respect to this work is that the scheme in  does not present a user with the plausible changes he/she may undertake to produce an effective counterexample, nor does it explain under what circumstances the user should impose their criterion over that of the model.
Iii Materials and Methods
This section describes the data and methods comprised in the proposed counterfactual generation architecture. To this end, first data and its variations will be explained, followed by a statement of the mini-max problem under consideration, as well as the methods devised for solving it efficiently.
Although the proposed framework could be extrapolated to any image classification task, explanations and experiments discussed throughout this work consider the so-called Celebrity dataset , which consists of celebrity face images with annotated facial attributes. This dataset is built by considering forty different facial attributes, from which thirteen are chosen to feed the GAN: Bald, Bangs, Black Hair, Blonde Hair, Brown Hair, Bushy Eyebrows, Eyeglasses, Gender, Mouth Open, Moustache, No Beard, Pale Skin and Age. For the sake of computational efficiency, the images in high definition contained in this dataset have been downsized to
, in which the first two numbers refer to the height and width in pixels of the scaled image, and the third number denotes the number of color channels. The downsizing procedure has been carried out by means of Tensorflow’s built-inbicubic resampling method. We adopt the original train, validation and test partitions established in , with sizes , and images, respectively. Further information on how these partitions have been made can be found in this reference.
Iii-B Problem Statement and Methods
The counterfactual generation framework proposed in this paper relies on a similar GAN architecture to the one introduced in . As shown in Figure 1, this architecture is composed by two models: one attempts to generate instances following certain characteristics (generator), whereas the other determines whether the generated instance belongs to a distribution of interest (discriminator). Originally the generator was fed with noise. More recently, other contributions [27, 28] have replaced the generator model with an encoder-decoder structure to allow for a semi-supervised training, by which the generated content will pertain to the distribution of the input rather than noise.
Several modifications have been imprinted to the conventional GAN structure to produce plausible counterfactuals of a third classification model. To begin with, following 
we have added an extra input vectorto the decoder towards biasing the output of the generator. This new vector
can target a manifold of purposes; however, for the task at hand (human face recognition), the vector permits to tailor what attributes the generator changes in the input face in order to fool the discriminator. By devising and introducing an additional loss termto the overall loss of the encoder-decoder GAN (with denoting the estimated vector of attributes of the reconstructed image by the decoder), we force the generator to maintain its input label at its output image, while changing certain facial attributes of the human face as per . Our addition to this architecture is non-intrusive: we include the classifier to be audited in parallel to the discriminator, yielding an additional loss term that quantifies the confidence of this classifier when producing its decision.
At this point it is important to emphasize that the audited model is left aside the overall training process of the GAN for several reasons. To begin with, we assume that for practicality, the access to the audited model must be kept at its minimum (black-box analysis). Therefore, we only exploit the logits of the audited model, with no further information on its inner structure whatsoever. Furthermore, the goal of the discriminator is to decide whether the generated image follows the distribution of the training set, which must be regarded as a plausibility check. The task undertaken by the audited model can be assorted, for instance, to discriminate amongmale and female, old and young or any other classification problem alike. Therefore, the attribute vector must balance between two objectives: 1) to maintain plausibility as per the discriminator, and 2) to confuse the audited model. A third objective can be also considered to account for the intensity of the changes inserted in the image as per : the subtler such changes are, the more unnoticed they will be in practice.
The above three goals can be formulated as a multi-objective optimization looking for a set of attribute vectors that produce, through the encoder-decoder of the GAN, images that achieve a Pareto-optimal balance between such goals. To recapitulate, one that determines if the discriminator is being fooled (reality check), another one that establishes whether the target fails to classify correctly the output of the GAN, and a third one that measures the intensity of attribute changes made to confuse the target model to be audited.
To mathematically state this problem, let be an image on which the counterfactual analysis is to be made, which follows distribution and has attributes . We denote the generator as , which is embodied by an encoder and a decoder , the latter receiving as its argument the compressed representation of the input image , and an attribute vector . The image output by the generator is given by . Ideally, , i.e. the reconstructed image should resemble itself.
A discriminator is placed next in the pipeline to determine i) whether the synthesized image is visually realistic, and ii) whether the attributes of the generated images correspond to those reflected in . To this end, we learn an attribute classifier from the annotated training image subset in order to compute an estimation of the attribute vector contained in its input image.
With all these ingredients being defined, and following 
, we can define the overall loss function that drives the learning algorithm of the encoder, decoder, discriminator and attribute classifier as a linear combination of the reconstruction and Wasserstein GAN losses, along with a loss reflecting that attributesshould be present in the reconstructed image. The training objective for encoder and decoder are given by:
where denotes expectation; indicates the distribution of possible attribute vectors in the range ; is the cross-entropy of binary distributions given by and ; and if is predicted to be a fake. When placing the focus on the discriminator and the classifier , the training goal is given by
where , and coefficients permit to balance the importance of the above losses during the training of the GAN architecture.
Once the GAN has been trained via back-propagation to meet the objectives in (1) and (5), we exploit this trained generative architecture to find counterfactual examples for a given test sample and the model being audited, which we hereafter refer to as , with classes . Specifically, we seek a minimum perturbation to the attribute vector (i.e. , with ) that, through and , yields a plausible image that succeeds at confusing the target model . The conflicting interplay between adversarial success, plausibility and intensity of the perturbation from which the counterfactual example is produced can be casted as a multi-objective optimization problem. Specifically,
quantifies the no-plausibility (unlikeliness) of the generated counterfactual through , which is given by the difference between the soft-max output of the discriminator corresponding to and (Wasserstein distance). The more negative this difference is, the more confident the discriminator is about the plausibility of the generated counterfactual ;
is the probability that the generated counterfactual does not confuse the target model , which is quantified by the negative value of the cross-entropy of the soft-max output of target model when queried with counterfactual ; and
gauges the intensity of adversarial changes made to the input image , which is given by .
To efficiently find a set of attribute perturbations balancing among the above three objectives in a Pareto-optimal fashion, we resort to multi-objective meta-heuristic optimization algorithms. Algorithm 1 summarizes the process of generating counterfactuals for target model , comprising both the training phase of the GAN architecture and the meta-heuristic search for counterfactuals subject to the three conflicting objectives stated above.
Iv Results and Discussion
As has been mentioned previously, we exemplify the application of our proposed plausible counterfactual generation framework by designing an experimental setup on the Celebrity dataset. Specifically, we aim at auditing a target classifier trained with this dataset to distinguish among classes (), which is to be done by fabricating perturbations of the attribute vector of the test image that maintain the plausibility of the produced counterfactual image . Two different experiments have been designed, in which we alter the data to train the classifier:
We use the entire training set to create a classifier that discriminates between Male and Female (i.e. the target variable is Gender).
We remove female examples with attribute Blonde hair from the training dataset, for reasons later disclosed throughout the discussion of the results.
For the process of finding counterfactual examples in the above experiments, we employ the recently contributed jMetalPy framework for multi-objective optimization with meta-heuristic search algorithms 
. This framework eases the process of defining custom multi-criteria optimization problems, and provides off-the-shelf optimization algorithms for efficiently discovering approximation of the Pareto front of the defined problems. In order to select the best performing algorithm, we have performed a benchmark comparison of three of them: NSGAII (Non-dominated Sorting Genetic Algorithm,), GDE3 (Generalized Differential Evolution, 
) and SMPSO (Speed-constrained Multi-objective Particle Swarm Optimization,). Hyper-parameters of these meta-heuristics have been tuned by means of an exhaustive search over a fine-grained grid of possible values, from where the best configuration was selected to be that yielding the best average hypervolume indicator value over 10 runs of every algorithm for images drawn at random from the test set. The reference point for the hypervolume was set to . This benchmark concluded that the best algorithm is SMPSO with a swarm size equal to particles, polynomial mutation with probability and archive size equal to individuals. Further information on this comparison study is not disclosed for a better focus of forthcoming explanations.
The target classifier
consists of a deep convolutional network that combines five sequential convolutional layers with batch normalization and leaky ReLu activations (rectified linear units with a 0.1 value under zero). This stacked set of convolutional layers end in a dense layer with a ReLu activation, followed by a second dense layer of a single unite activated by a sigmoid function that outputs the probability of being of the target class. This model has been trained separately, and added to the overall architecture once the weights are loaded from the solution published in, namely, a fully-functional, pre-trained attribute-based GAN able to generate facial attributes as desired. Some modifications have been made to the architecture and the loss functions to incorporate the target classifier to be audited. Therefore, the GAN training step of Algorithm 1 (line 1) has not been needed to produce the results discussed in the following subsections.
Iv-a Experiment 1: Target Classifier trained with Entire Data
Figures 2 and 3 summarize the results obtained from the first experiment. On the one hand, Figure 2.a depicts the Pareto front achieved for test images corresponding to male and female images, whereas Figure 2.b shows the progression of the probability from the target classifier corresponding to the male class, overlaid by the values of the and of every point in the estimated Pareto front of Figure 2.a. Counterfactuals are arranged in decreasing (male images) and decreasing order (female images) of the probability of class male output by .
We comment on this first plot before proceeding further. To begin with, the intensity of changes to audit model when the test image corresponds to a female human are in general more intense (i.e. higher value of ). It is also interesting to see that, for both genders, a crossing point can be noted in Figure 2.b between and . As shown next, this point determines the best plausible counterfactual corresponding to each test image.
We proceed with the discussion of the results of the first experiment by inspecting Figure 3 (next page), which illustrates the generated images from the perturbation vectors of the Pareto front depicted in Figure 2.a. For the sake of space, only every tenth of the counterfactuals sorted as per Figure 2.b are shown, thereby unfolding a representative excerpt of the diversity of images synthesized by our proposed framework. It is surprisingly accurate and recurrently happening over many examples in the test set that the aforementioned crossing point between and corresponds to the most plausible counterfactual, as can be verified by simple visual inspection. Furthermore, a similar exploration of the counterfactuals produced by many other images unveils that blonde hair is recurrently added to male images so as to make them be classified as female by , and vice versa: darkened hair color is imprinted to female images to make them be predicted as male.
Iv-B Experiment 2: Target Classifier trained with Skewed Data
The previous experiment elucidates that the optimization process tends to turn the hair of female images to black, and the hair of male images to blond. There lies the rationale for devising a second experiment, where all images of female faces with blonde hair are removed from the training dataset of the target model. Figure 4 shows two produced series of counterfactuals for this second experiment. In the case of the original male image (upper pair of plots), blonde haired counterfactuals are eventually produced, yet plummets even if the hair is not blonde. A similar effect is perceived in the other depicted case, where the conversion of a female to a male realistically is attained by modifying other image attributes that are not necessarily related to the color of the hair (e.g. lower bang, higher Age) to increase . Although the color of the hair appears to still influence the production of counterfactuals for the female case, it is not as evident as in the case of Experiment 1. This corroborates the potential of our framework for discovering hidden sources of bias in classification problems that are not properly detected nor removed during the modeling phase.
V Conclusions and Future Research Lines
This work has elaborated on adversarial learning framework oriented towards the multi-objective synthesis of adversarial samples aimed at explaining the performance limits of Deep Learning classification problems. Unlike other adversarial learning studies wherein the goal is to produce subtle modifications to the input of the model, we aim at imprinting plausible changes that the user of the image could realistically imprint in the input to the model, so that plausibility is enforced to the produced counterfactual. To this end, we have focused on image classification, and hybridized an attribute GAN model  with a multi-objective meta-heuristic search engine to achieve counterfactuals for a given image that balance between three objectives related to the explainability of the classifier: plausibility, adversarial success and intensity of the attribute perturbation. Two experiments have elicited interesting conclusions that go beyond the initially targeted goal of this study. Specifically, we have proven that the proposed framework may serve well to check underlying biasing phenomena present in a model and/or a dataset. Indeed, a subtle bias of the dataset (more blond women) was shown to be crucial for the adversarial success of male and female counterfactuals, informing the user about the risk of not tackling it (i.e. a simple change of color hair could eventually make a test image be classified wrongly by the audited model).
In light of the global concern with the accountability of Artificial Intelligence methods, we plan to invest further efforts along several research directions rooted on the findings reported in this work. To cite a few, a closer look will be taken at how the counterfactual information generated by our framework can be fed back to the audited model and increase its robustness against adversarial or, alternatively, counteract the presence of bias in the training data. From the algorithmic side, Reinforcement Learning algorithms will be explored as an efficient alternative to multi-objective meta-heuristics, capable of learning the relationship between attribute changes and multi-criteria effect on the test image under consideration.
The authors would like to thank the Basque Government for its support through the EMAITEK and ELKARTEK funding programs. Javier Del Ser also receives support from the Consolidated Research Group MATHMODE (IT1294-19) granted by the Department of Education of the Basque Government.
-  A. Kamilaris and F. X. Prenafeta-Boldú, “Deep learning in agriculture: A survey,” Computers and Electronics in Agriculture, vol. 147, pp. 70–90, 2018.
-  J. Del Ser, E. Osaba, J. J. Sanchez-Medina, and I. Fister, “Bioinspired computational intelligence and transportation systems: a long road ahead,” IEEE Transactions on Intelligent Transportation Systems, 2019.
A. Diez-Olivan, J. Del Ser, D. Galar, and B. Sierra, “Data fusion and machine learning for industrial prognosis: Trends and perspectives towards industry 4.0,”Information Fusion, vol. 50, pp. 92–111, 2019.
-  N. D. Lane and P. Georgiev, “Can deep learning revolutionize mobile sensing?” in 16th ACM International Workshop on Mobile Computing Systems and Applications, 2015, pp. 117–122.
-  M. A. Alsheikh, D. Niyato, S. Lin, H.-P. Tan, and Z. Han, “Mobile big data analytics using deep learning and Apache Spark,” IEEE Network, vol. 30, no. 3, pp. 22–29, 2016.
-  Z. Yuan, Y. Lu, Z. Wang, and Y. Xue, “Droid-sec: deep learning in android malware detection,” in ACM SIGCOMM Computer Communication Review, vol. 44, no. 4, 2014, pp. 371–372.
-  I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Adv. in Neural Information Processing Systems, 2014, pp. 2672–2680.
-  M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein generative adversarial networks,” in International Conference on Machine Learning, 2017, pp. 214–223.
-  N. Papernot and P. McDaniel, “Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning,” arXiv preprint arXiv:1803.04765, 2018.
-  Y. Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in International Conference on Machine Learning, vol. 48, 2016, pp. 1050–1059.
-  A. Subramanya, S. Srinivas, and R. V. Babu, “Confidence estimation in deep neural networks via density modelling,” arXiv preprint arXiv:1707.07013, 2017.
-  M. D. Zeiler, D. Krishnan, G. W. Taylor, and R. Fergus, “Deconvolutional networks,” in
-  K. Simonyan, A. Vedaldi, and A. Zisserman, “Deep inside convolutional networks: Visualising image classification models and saliency maps,” arXiv preprint arXiv:1312.6034, 2013.
-  S. Bach, A. Binder, G. Montavon, F. Klauschen, K.-R. Müller, and W. Samek, “On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation,” PloS one, vol. 10, no. 7, p. e0130140, 2015.
-  A. Barredo Arrieta, N. Díaz-Rodríguez, J. Del Ser, A. Bennetot, S. Tabik, A. Barbado, S. García, S. Gil-López, D. Molina, R. Benjamins, and F. Herrera, “Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI,” Information Fusion, vol. 58, pp. 82–115, 2020.
-  A. Hindupur, “The GAN zoo: A list of all named GANs,” 2017.
-  H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. N. Metaxas, “Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks,” in IEEE International Conference on Computer Vision, 2017, pp. 5907–5915.
-  H. Wu, S. Zheng, J. Zhang, and K. Huang, “Gp-gan: Towards realistic high-resolution image blending,” in ACM International Conference on Multimedia, 2019, pp. 2487–2495.
-  Z. He, W. Zuo, M. Kan, S. Shan, and X. Chen, “Attgan: Facial attribute editing by only changing what you want,” IEEE Transactions on Image Processing, vol. 28, no. 11, pp. 5464–5478, 2019.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang,
A. Karpathy, A. Khosla, M. Bernstein et al.
, “Imagenet large scale visual recognition challenge,”International Journal of Computer Vision, vol. 115, no. 3, pp. 211–252, 2015.
-  A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Adv. in Neural Information Processing Systems, 2012, pp. 1097–1105.
-  Z. C. Lipton, “The mythos of model interpretability,” Queue, vol. 16, no. 3, pp. 31–57, 2018.
-  M. T. Ribeiro, S. Singh, and C. Guestrin, “Why should i trust you?: Explaining the predictions of any classifier,” in ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016, pp. 1135–1144.
-  S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” in Adv. in Neural Information Processing Systems, 2017, pp. 4765–4774.
-  S. Liu, B. Kailkhura, D. Loveland, and Y. Han, “Generative counterfactual introspection for explainable deep learning,” arXiv preprint arXiv:1907.03077, 2019.
-  Z. Liu, P. Luo, X. Wang, and X. Tang, “Deep learning face attributes in the wild,” in International Conference on Computer Vision, 2015, pp. 3730–3738.
-  S. Arora, A. Risteski, and Y. Zhang, “Theoretical limitations of encoder-decoder GAN architectures,” arXiv preprint arXiv:1711.02651, 2017.
-  P. Lyu, X. Bai, C. Yao, Z. Zhu, T. Huang, and W. Liu, “Auto-encoder guided GAN for chinese calligraphy synthesis,” in IEEE International Conference on Document Analysis and Recognition, vol. 1, 2017, pp. 1095–1100.
A. Benitez-Hidalgo, A. J. Nebro, J. Garcia-Nieto, I. Oregi, and J. Del Ser,
“jMetalPy: a Python framework for multi-objective optimization with
Swarm and Evolutionary Computation, vol. 51, p. 100598, 2019.
-  K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan, “A fast and elitist multiobjective genetic algorithm: NSGA-II,” IEEE Transactions on Evolutionary Computation, vol. 6, no. 2, pp. 182–197, 2002.
-  S. Kukkonen and J. Lampinen, “GDE3: The third evolution step of generalized differential evolution,” in IEEE Congress on Evolutionary Computation, vol. 1, 2005, pp. 443–450.
-  A. J. Nebro, J. J. Durillo, J. Garcia-Nieto, C. C. Coello, F. Luna, and E. Alba, “SMPSO: A new PSO-based metaheuristic for multi-objective optimization,” in IEEE Symposium on Computational Intelligence in Multi-Criteria Decision-Making, 2009, pp. 66–73.