Log In Sign Up

Two Gaussian Approaches to Black-Box Optomization

by   Lukas Bajer, et al.

Outline of several strategies for using Gaussian processes as surrogate models for the covariance matrix adaptation evolution strategy (CMA-ES).


page 1

page 2

page 3

page 4


Adaptive Generation-Based Evolution Control for Gaussian Process Surrogate Models

The interest in accelerating black-box optimizers has resulted in severa...

Landscape Analysis for Surrogate Models in the Evolutionary Black-Box Context

Surrogate modeling has become a valuable technique for black-box optimiz...

Evolution Attack On Neural Networks

Many studies have been done to prove the vulnerability of neural network...

Using CMA-ES for tuning coupled PID controllers within models of combustion engines

Proportional integral derivative (PID) controllers are important and wid...

The Hessian Estimation Evolution Strategy

We present a novel black box optimization algorithm called Hessian Estim...

1 CMA Evolution Strategy

CMA-ES [7, 8]

is the state-of-the-art evolutionary optimization method, at least in the area of continuous black-box optimization. Basically, it consists in generating new search points by sampling from a multidimensional normal distribtion, the mean and variance of which are updated from generation to generation. In particular, the population

of the -st generation,

, follows the normal distribution with mean

and variance resulting from the update in the -th generation,


Here, is the step size in the -th generation. and are updated separately,

being in the most simple case obtained as the unbiased empirical estimate based on the

-th generation:


The positive semidefinite matrix

can be diagonalised in the basis formed by its eigenvectors



where and is a diagonal matrix such that its

-th diagonal element is the eigenvalue corresponding to the eigenvector

. Consequently, the coordinates of points generated according to (1) in that basis, , are uncorrelated:


2 Optimization Based on Gaussian Processes

A Gaussian process (GP) on a -dimensional Euclidean space

is a collection of random variables, GP

, such that the joint distribution of any finite number of them is a multidimensional normal distribtion. Following

[4], we differentiate two ways of using GPs in black-box optimization:

  • As a surrogate model to be optimized instead of the black box objective function. On the found optimum or optima, the original black-box objective function is then evaluated. This use of GPs has been introduced in the popular Efficient Global Optimization (EGO) algorithm [14]. As the optimization method, also evolutionary optimization can be used, in particular CMA-ES [4], but this is not the only possibility: For example, traditional low-degree polynomial models, aka response surface models [20], are much more efficiently optimized using traditional smooth optimization methods.

  • For the evolution control of the evolutionary optimization of the original objective function, i.e., for controlling the composition of its population [5, 6, 9, 23].

Irrespective of the way in which a GP is used, its construction is always based on a sequence of training pairs and is subsequently employed to compute the random variable for . Since the distribution of is multidimensional normal, also the conditional distribution of conditioned on is normal,


where . There are two commonly encountered possibilities how define the functions , describing the conditional GP mean, and , describing the conditional GP variance:

  1. A GP is the superposition of a deterministic function and a GP with zero mean. For the latter, it can be shown [21] that the function describing its mean fulfils


    whereas the function describing its variance fulfills


    Here, an i.i.d. Gaussian noise is assumed, is the

    -dimensional identity matrix,

    is a symmetric function, and


    The resulting superposition then fulfils


    whereas .

  2. A GP is the superposition of a Bayesian mean assuming the multiplicative form and a GP with zero mean, where is a random variable with and has the same meaning as above. In that case, it can be shown [21] that for ,




Simple, but frequently used examples of the function occurring in (6)–(7) and (10)–(11) include:

  • Squared exponential,


    Here, is a parameter called characteristic length-scale. It is a parameter of the function , not of the Gaussian process, the process is nonparametric. Therefore is referred to as a hyperparameter.

  • -Exponential,


    which has 2 hyperparameters, and .

  • Dot product,


    which has 2 hyperparameters, and , or in its generalized version,


    which has, in addition, the matrix of hyperparameters , defining a dot product in general coordinates.

2.1 GP-based criteria to choose the points for evaluation

Whereas traditional response surface and surrogate models employ basically only one criterion for the choice of points in which the black box objective function should be evaluated, namely the global or at least local minimum of the model (if the optimization objective is minimization), GPs offer several additional criteria:

  • Minimum of a prescribed quantile (Q

    ) of the distribution of , .


    Usually, (18) is expressed using quantiles of the standard normal distribution, ,


    For , (19) turns to the traditional global minimum criterion, applied to .

  • Probability of improvement (PoI),


    where denotes the distribution function of , is the minimum value found so far. More generally, probability of improvement with respect to a given ,

  • Expected improvement (EI),


    Introducing the normalized mean improvement ,


    and the notation for the density of , (22) can be expressed as [13]


3 Possible Synergy

Directly connecting both considered Gaussian approaches is not possible because the normal distribution in CMA-ES is a distribution on the input space of the objective function (fitness), whereas the normal distribution in GPs is on the space of its function values. Nevertheless, it is still possible to achieve some synergy through using information from CMA-ES for the GP, and/or using information from the GP for CMA-ES. According to whoat was recalled at the beginning of Section 2, the latter possibility corresponds to using GP for the evolution control of CMA-ES.

3.1 Using Information from CMA-ES for the GP

We see 2 straightforward possibilities where some information from CMA-ES can be used in the GP.

  1. The function occurring in (9), (10) and (11) can be constructed using the fitness values of individuals from some particular generation or several generations of CMA-ES. Its construction can be as simple as setting to a constant aggregating the considered fitness values, e.g., their mean or weighted mean, but it can also consist in training, with those values, a response surface model, principally of any kind.

  2. Kruisselbrink et al. [16], who combine CMA-ES with a GP using the -exponential function , propose to employ in (15

    ) the Mahalanobis distance of vectors

    and given by the covariance matrix obtained in the -th generation of CMA-ES, , instead of their Euclidean distance. This is actually a specific consequence of another possibility of using information from CMA-ES for the GP – replacing the original space of -dimensional vectors, , by the space of their principal components with respect to . In this context, it is worth recalling that in [2, 3, 15], Mahalanobis instead of Euclidean distance was used when combining CMA-ES with quadratic response surface models in [2, 3, 15]. In the approach presented there, the space of the principal components with respect to is used, together with an estimate of density, to locally weight the model predictions with respect to the considered input. In this way, a connection to the other kind of synergy is established, i.e., to the evolution control of CMA-ES, giving us the possibility to use both kinds in combination.

3.2 GP-Based Evolution Control of CMA-ES

We intend to test the following approaches to using GP for the evolution control of CMA-ES.

Algorithm 1
Input: generation , points , their linear ordering with according to a GP-based criterion to choose the points for evaluation, . Step 1. Perform -means clustering of into sets . Step 2. For , choose . Step 3. For , choose . Output: Points to be evaluated by the original fitness function.
Figure 1: Algorithm of the proposed strategy for choosing the points to be evaluated by the original fitness from among the evaluated by the Gaussian process
  • Basic approach. In the -th generation, points are sampled from the distribution , where is several to many times larger then . For all of them, a selected criterion from among those introduced in 2.1 is computed. Based on the value of that criterion, the points for the evaluation by the original black-box fitness are chosen. This can be done according to various strategies, we intend to use the one described in Algorithm 1

    , with which we have a good experience from using radial basis function networks for the evolution control in the evolutionary optimization of catalytic materials

    [10]. The linear ordering stands in the cases PoI and EI for , in the case for . The use of a linear ordering relates the proposed approach to ranking-based evolution control of CMA-ES [2, 3, 15, 23], as well as to surrogate modelling of CMA-ES by ordinal regression [17, 18, 19, 22]. Most similar is the approach by Ulmer et al. [23], the difference being that they don’t use clustering.

  • GP on low-dimensional projections attempts to improve the basic approach in view of the experience reported in the literature [15] and obtained also in our earlier experiments [1] that GPs are actually advantageous only in low dimensional spaces. Instead of the sampled points , the GP is trained only with their first principal components with respect to , , i.e., with the projections of to the -dimensional space , provided the orthonormal eigenvectors are enumerated according to decreasing eigenvalues.

  • GP on low-dimensional projections within restricted distance attempts to decreaase the deterioration of the GP on low-dimensional projections with respect to the GP on the original sampled points by using only points within a prescribed small distance from their respective projections . To this end, points from the distribution are resampled until points are obtained fulfilling

  • Two-stage sampling. Trainig the GP for the -th generation can easily suffer from the lack of training data in a part of the search space where a substantial proportion of the points will be sampled. To alleviate it, the points to be evaluated by the original fitness can be sampled in two stages:

    1. First, points , where , are sampled from the distribution , evaluated by the original fitness, and included into training the GP.

    2. Then, points are sampled from the same distribution, and for them, a selected criterion from among those introduced in 2.1 is computed, based on which the points for the evaluation by the original black-box fitness are chosen. To this end, again Algorithm 1 can be used, with the following changes:

      • In the input, the numbers and have now to fulfill

      • In Step 3, is chosen only for .

      • The output contains only the points .

    Needless to say, this approach has to be combined with some of the approaches (i)–(iii) and can be combined with any of them.

  • Generation-based evolution control. Whereas the previous four approaches represent, in terms of [11, 12], individual-based evolution control, we want to test also one approach that is generation-based, in the sense that the desired number of points is evaluated by the original black-box fitness function only in selected generations. Similarly to the evolution strategy in [18, 19], our approach selects those generations adaptively, according to the agreement between the ranking of considered points by the surrogate model and by the black-box fitness. Differently to that strategy, however, we want to base the estimation if that agreement not on the generations in which points have been evaluated by the black-box fitness, but on evaluating a small and evolvable number of additional points in each generation. In this context, it is important that in situations when the fitness is evaluated empirically, using some measurement or testing, the evaluation hardware causes the evaluation costs to increase step-wise, and to remain subsequently constant for some evaluated points (e.g. in the optimization of catalyst preformance described in [10], is the number of channels in the chemical reactor in which the catalysts are tested). In such a situation, the costs of evaluation are the lowest if is a multiple of , and if the evaluation of the additional points in each generation is cumulated for generations such that


    Setting covers the case when such a situation does not occur and the evaluation costs increase linearly with the number of points. If is the last generation in which points have been evaluated by the black-box fitness, then for , points are sampled from the distribution , from which the points are selected by Algorithm 1, in which the input is replaced with . Subsequently, all the points are evaluated by the black-box fitness, and for each generation , the agreement between the ranking of by the current Gaussian process, GP, and the black-box fitness is estimated. If the agreement is sufficient in all considered generations, then the procedure is repeated using instead of and unchanged . Otherwise, additional points are selected from for the first generation in which the agreement was not sufficient. Then a new Gaussian process, GP, is trained using the training set


    optionally including also some or all points from the set used for training GP. At that occasion, aslo the value of can be changed.

    Provided the conditions and are fulfilled, the value of can be arbitrary. Needless to say, the smaller , the larger will be the proportion of points from the generation in which the GP was trained in its training set, and the more similar the obtained GP will normally be to a GP trained only with data from that generation, which is the usual way of using surrogate models in traditional generation-based strategies [11, 12, 18, 19]. It is also worth pointing out that our generation-based evolution control was explained here as a counterpart to the basic individual-based approach (i), but counterparts to the approaches (ii) and (iii) are possible as well.


  • [1] L. Bajer, V. Charypar, and M. Holeňa. Model guided sampling optimization with Gaussian processes for expensive black-box optimization. In GECCO 2013, pages 1715–1716, 2013.
  • [2] Z. Bouzarkouna, A. Auger, and D.Y. Ding. Investigating the local-meta-model cma-es for large population sizes. In EvoApplications 2010, pages 402–411, 2010.
  • [3] Z. Bouzarkouna, A. Auger, and D.Y. Ding. Local-meta-model cma-es for partially separable functions. In GECCO 2011, pages 869–876, 2011.
  • [4] D. Büche, N.N. Schraudolph, and P. Koumoutsakos.

    Accelerating evolutionary algorithms with Gaussian process fitness function models.

    IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, 35:183–194, 2005.
  • [5] M.A. El-Beltagy and A.J. Keane. Evolutionary optimization for computationally expensive problems using Gaussian processes. In

    International Conference on Artificial Intelligence

    , pages 708–714, 2001.
  • [6] M. Emmerich, A. Giotis, M. Özdemir, T. Bäck, and K. Giannakoglou. Metamodel-assisted evolution strategies. In PPSN VII, pages 361–370, 2002.
  • [7] N. Hansen. The CMA evolution strategy: A comparing review. In

    Towards a New Evolutionary Computation

    , pages 75–102. Springer Verlag, Berlin, 2006.
  • [8] N. Hansen and A. Ostermaier. Completely derandomized self-adaptation in evolution strategies. Evolutionary Computation, 9:159–195, 2001.
  • [9] E. Hoffman and S. Holemann. Controlled model assisted evolution strategy with adaptive preselection. In International IEEE Symposium on Evolving Fuzzy Systems, pages 182–187, 2006.
  • [10] M. Holeňa, D. Linke, and L. Bajer. Surrogate modeling in the evolutionary optimization of catalytic materials. In GECCO 2012, pages 1095–1102, 2012.
  • [11] Y. Jin. A comprehensive survery of fitness approximation in evolutionary computation. Soft Computing, 9:3–12, 2005.
  • [12] Y. Jin, M. Olhofer, and B. Sendhoff. Managing approximate models in evolutionary aerodynamic design optimization. In CEC 2001, pages 592–599, 2001.
  • [13] D.R. Jones. A taxonomy of global optimization methods based on response surfaces. Journal of Global Optimization, 21:345–383, 2001.
  • [14] D.R. Jones, M. Schonlau, and W.J. Welch. Efficient global optimization of expensive black-box functions. Journal of Global Optimization, 13:455–492, 1998.
  • [15] S. Kern, N. Hansen, and P. Koumoutsakos. Local metamodels for optimization using evolution strategies. In PPSN IX, pages 939–948, 2006.
  • [16] J.W. Kruisselbrink, M.T.M. Emmerich, A.H. Deutz, and T. Bäck. A robust optimization approach using kriging metamodels for robustness approximation in the CMA-ES. In CEC 2010, pages 1–8, 2010.
  • [17] I. Loshchilov, M. Schoenauer, and M. Seba. Comparison-based optimizers need comparison-based surrogates. In PPSN XI, pages 364–373, 2010.
  • [18] I. Loshchilov, M. Schoenauer, and M. Seba. Self-adaptive surrogate-assisted covariance matrix adaptation evolution strategy. In GECCO 2012, pages 321–328, 2012.
  • [19] I. Loshchilov, M. Schoenauer, and M. Seba. Intensive surrogate model exploitation in self-adaptive surrogate-assisted CMA-ES (saACM-ES). In GECCO 2013, pages 439–446, 2013.
  • [20] R.H. Myers, D.C. Montgomery, and C.M. Anderson-Cook. Response Surface Methodology: Proces and Product Optimization Using Designed Experiments. John Wiley and Sons, Hoboken, 2009.
  • [21] E. Rasmussen and C. Williams.

    Gaussian Process for Machine Learning

    MIT Press, Cambridge, 2006.
  • [22] T.P. Runarsson. Ordinal regression in evolutionary computation. In PPSN IX, pages 1048–1057, 2006.
  • [23] H. Ulmer, F. Streichert, and A. Zell. Evolution strategies assisted by Gaussian processes with improved pre-selection criterion. In IEEE Congress on Evolutionary Computation, pages 692–699, 2003.