There is a limited literature on the multi-step lookahead problem. Osborne et al. (2009) perform multi-step lookahead by optimising future evaluation locations, and sampling over future function values. This approach scales poorly with the number of future evaluations considered, and the authors present results for no more than two-step lookahead. (Marchant et al., 2014)
reframe the multi-step lookahead problem as a partially observed Markov decision process, and adopt a Monte Carlo tree search approach in solving it. Again, the scaling of the approach permits the authors to consider no more than six steps into the future. In the past, the multi-step look ahead problem was studied byStreltsov and Vakili (1999) proposing a utility function that maximizes the total ‘reward’ of the algorithm by taking into account the cost of future computations, rather than trying to find the optimum after a fixed number of evaluations.
Interestingly, there exists a link between the multi-step lookahead problem and batch Bayesian optimisation (Ginsbourger et al., 2009; Azimi et al., 2012). In this later case, batches of locations rather than individual observations are selected in each iteration of the algorithm and evaluated in parallel. When such locations are selected greedily, that is, one after the other, the key to selecting good batches relies on the ability of the batch criterion of predicting future steps of the algorithm. In this work we will exploit this parallelism to compute a non-myopic loss for Bayesian optimisation.
This paper is organised as follows. In Section 2 we formalise the problem and describe the contributions of this work. Section 3 describe the details of the proposed algorithm. Section 4 illustrates the superior performance of glasses in a variety of test functions and we conclude in Section 5 with a discussion about the most interesting results observed in this work.
2 Background and challenge
2.1 Bayesian optimisation with one step look-ahead
Let be well behaved function defined on a compact subset . We are interested in solving the global optimization problem of finding
We assume that is a black-box from which only perturbed evaluations of the type , with , are available. Bayesian Optimization (bo
) is an heuristic strategy to make a series of evaluationsof , typically very limited in number, such that the the minimum of is evaluated as soon as possible (Lizotte, 2008; Jones, 2001; Snoek et al., 2012; Brochu et al., 2010).
Assume that points have been gathered so far, having a dataset . Before collecting any new point, a surrogate probabilistic model for is calculated. This is typically a Gaussian Process (gp) with mean function and a covariance function , and whose parameters will be denoted by . Let be the current available information: the conjunction of , the model parameters and the model likelihood type. Under Gaussian likelihoods, the predictive distribution for at
is also Gaussian with mean posterior mean and variance
where is the matrix such that , (Rasmussen and Williams, 2005).
Given the gp model, we now need to determine the best location to sample. Imagine that we only have one remaining evaluation () before we need to report our inferred location about the minimum of . Denote by , the current best found value. We can define the loss of evaluating this last time at assuming it is returning as
Its expectation is
where the subscript in refers to the fact that we are considering one future evaluation. Giving the properties of the gp, can be computed in closed form for any . In particular, for
the usual Gaussian cumulative distribution function, we have that
where we have abbreviated as and as . Finally, the next evaluation is located where gives the minimum value. This point can be obtained by any gradient descent algorithm since analytical expressions for the gradient and Hessian of exist (Osborne, 2010).
2.2 Looking many steps ahead
Expression (2.1) can also be used as a myopic approximation to the optimal decision when evaluations of remain available. Indeed, most bo methods are myopic and ignore the future decisions that will be made by the algorithm in the future steps.
Let for be the remaining available evaluations and by the available information after the data set has been augmented with and the parameters of the model updated. We use to denote the expected loss of selecting given and considering future evaluations. A proper Bayesian formulation allows us to define this long-sight loss (Osborne, 2010) as111We assume that .
is the predictive distribution of the gp at and
reflects the optimization step required to obtain after all previous the evaluations have been iteratively optimized and marginalized. The graphical probabilistic model underlying (2) is illustrated in Figure 1.
To evaluate Eq. (2) we can successively sample from to and optimize for the appropriate . This is in done in (Osborne, 2010) for only two steps look ahead. The reason why further horizons remain unexplored is the computational burden required to compute this loss for many steps ahead. Note that analytical expression are only available in the myopic case .
2.3 Contributions of this work
The goal of this work is to propose a computationally efficient approximation to Eq. (2) for many steps ahead able to relieve the myopia of classical Bayesian optimization. The contributions of this paper are:
A new algorithm, glasses, to relieve the myopia of Bayesian optimisation that is able to efficiently take into account dozens of steps ahead. The method is based on the prediction of the future steps of the myopic loss to efficiently integrate out a long-side loss.
The key aspect of our approach is to split the recursive optimization marginalization loop in Eq. (2) into two independent optimisation-marginalization steps that jointly act on all the future steps. We propose an Expectation-Propagation formulation for the joint marginalisation and we discuss different strategies to carry out the optimisation step.
Together with this work, we deliver a open source Python code framework222http://sheffieldml.github.io/GPyOpt/ containing a fully functional implementation of the method useful to reproduce the results of this work and applicable in general global optimisation problems. As we mentioned in the introduction of this work, there exist a limited literature in bo non-myopic methods and, to our knowledge, none available generic bo package can be used with myopic loss functions.
Simulations: New practical experiments and insights that show that non-myopic methods outperform myopic approaches in a benchmark of optimisation problems.
3 The GLASSES algorithm
As detailed in the previous section, a proper multi-step look ahead loss function requires the iterative optimization-marginalization of the future steps, which is computationally intractable. A possible way of dealing with this issue is to jointly model our epistemic uncertainty over the future locations
with a joint probability distributionand to consider the expected loss
the vector of future evaluations ofand X the dimensional matrix whose rows are the future evaluations . is multivariate Gaussian, since it corresponds to the predictive distribution of the gp at X. The graphical probabilistic model underlying (2) is illustrated in Figure 2. differs from in the fact that all future evaluations are modelled jointly rather then sequentially. A proper choice of is crucial here. An interesting option would be to choose to be some determinantal point process (dpp)333 A determinantal point process is a probability measure over sets that is entirely characterised by the determinant of some (kernel) function. defined on (Affandi et al., 2014) and integrate Eq. (3) with respect to by averaging over multiple samples (Kulesza and Taskar, 2012, 2011). DPPs provide a density over sample locations that induces them to be dissimilar to each other (as well-spaced samples should), but that can be concentrated in chosen regions (such as regions of low myopic expected loss). However, although DPPs have nice computational properties in discrete sets, here we would need to take samples from the dpp by conditioning on and the number of steps ahead. Although this is possible in theory, the computational burden of generating these samples will make this strategy impractical.
An alternative and more efficient approach, that we explore here, is to work with a fixed X, which we assume it is computed beforehand. As we show in this section, although this approach does not make use of our epistemic uncertainty on the future steps, it drastically reduces the computational burden of approximating .
3.1 Oracle multiple steps look-ahead expected loss
Suppose that we had access to an oracle function able to predict the future locations that the loss would suggest if we started evaluating at . In other words, takes the putative location as input and it returns and the predicted future locations . We work here under the assumption that the oracle has perfect information about the future locations, in the same way we have have perfect information about the locations that the algorithm already visited. This is an unrealistic assumption in practice, but it will help us to set-up our algorithm. We leave for the next section the details of how to marginalise over .
Assume, for now, that exists and that we have access to it. We it and denote by the vector of future locations evaluations of at . Assuming that is known, it is possible to rewrite the expected loss in Eq. (2) as
where the expectation is taken over the multivariate Gaussian distribution, with mean vectorand covariance matrix , that gives rise after marginalizing the posterior distribution of the GP at . Note that under a fixed , it also holds that . See supplementary materials for details.
The intuition behind Eq. (4) is as follows: the expected loss at is the best possible function value that we expect to find in the next steps, conditional on the first evaluation being made at . Rather than merely quantifying the benefit provided by the next evaluation, this loss function accounts for the expected gain in the whole future sequence of evaluations of . As we analyse in the experimental section of this work, the effect of this is an adaptive loss that tends to be more explorative when more remaining evaluations are available and more exploitative as soon as we approach the final evaluations of .
See supplementary materials for details. The first term in Eq. (3.1) is a Gaussian probability on an unbounded polyhedron in which the limits are aligned with the axis. The second term is the sum of the Gaussian expectations on different non-axis-aligned different polyhedra defined by the indicator functions. Both terms can be computed with ep using the approach proposed in (Cunningham et al., 2011). In a nutshell, to compute the integrals it is possible to replace the indicator functions with univariate Gaussians that play the role of soft-indicators in the ep iterations. This method is computationally efficient and scales well for high dimensions. Note that when , Eq. (4) reduces to Eq. (2.1).
Under the hypothesis of this section, the next evaluation is located where gives the minimum value. We still need, however, to propose a way to approximate the oracle . We do in next section.
3.2 Local Penalisation to Predicting the Steps Ahead
This section proposes an empirical surrogate for . A sensible option would be to use the maximum a posteriori probability, map, of the above-mentioned dpp. However, as it is the generation of dpp samples, to compute the map of a dpp is an expensive operation (Gillenwater et al., 2012). Alternatively, here we use some ideas that have been recently developed in the batch Bayesian optimisation literature. In a nutshell, batch bo methods aim to define sets of points in where should be evaluated in parallel, rather than sequentially. In essence, a key aspect to building a ‘good’ batch is the same as to computing a good approximation for : to find a set of good locations at which to evaluate the objective.
In this work we adapt to our context the batch bo idea proposed by González et al. (2015). Inspired by the repulsion properties of dpp, González et al. (2015) propose to build each batch by iteratively optimising and penalising the acquisition function in a neighbourhood of the already collected points by means of some local penalisers . Note that any other batch method could be used here, but we consider this approach since it is computationally tractable and scales well with the size of the batches (steps ahead in our context) and the dimensionality of the problem.
More formally, assume that the objective function is -Lipschitz continuous, that is, it satisfies that , . Take and valid Lipschitz constant . It is possible to show that the ball
where , doesn’t contain the minimum of . Probabilistic versions of these balls are used to define the above mentioned penalisers by noting that . In particular, is chosen as the probability that , any point in that is a potential candidate to be a minimum, does not belong to : As detailed in González et al. (2015), these functions have a closed form and create an exclusion zone around the point . The predicted k-th location when looking at step ahead and using as putative point is defined as
for and where are local local penalizers centered at and is the soft-plus transformation .
To illustrate how computes the steps ahead we include Figure (3). We show the myopic loss in a bo experiment together with predicted locations by for two different putative points. In case 1, the putative point (grey star) is close to the location of the minimum of the myopic loss (blue region). In case 2, is located in an uninteresting region. In both cases the future locations explore the interesting region determined for the myopic loss while the locations of the points are conditioned to the first location. In the bottom row of the figure we show how the points are selected in Case 1. The first putative point creates an exclusion zone that shifts the minimum of the acquisition, where the next location is selected. Iteratively, new locations are found by balancing diversity (due to the effect of the exclusion areas) and quality (exploring locations where the loss is low), similarly to the way samples the probabilities over subsets can be characterised in a dpp (Kulesza and Taskar, 2012).
3.3 Algorithm and computational costs
All the steps of glasses are detailed in Algorithm 1. The main computational cost is the calculation of the steps ahead, which is done using a sequence of l-bfgs optimizers at complexity for , the maximum number of l-bfgs updates. The use of ep to compute the value of the expected loss at each location requires a quadratic run time factor update in the dimensionality of each Gaussian factor.
4.1 Interpreting the non-myopic loss
The goal of this experiment is to visualise the effect on the expected loss of considering multiple steps ahead. To this end, we use six-hump camel function (see Table 4 for details). We fit a gp with a square exponential kernel and we plot the myopic expected loss together with 5 variants that consider 2, 3, 5, 10 and 20 steps ahead. Increasing the steps ahead decreases the optimum value of the loss: the algorithm can visit more locations and the expected minimum is lower. Also, increasing the steps ahead flattens down the loss: it is likely to hit a good location irrespective of the initial point so all candidate looks better because of the future chances of the algorithm to be in a good location. In practice this behaviour translates into an acquisition function that becomes more explorative as we look further ahead.
4.2 Testing the effect of considering multiple steps ahead
|Alpine2-||2, 5, 10|
To study the validity of our approximation we choose a variety of functions with a range of dimensions and domain sizes. See Table 1 for details. We use the full glasses algorithm (in which at each iteration the number of remaining iterations is used as the number of steps-ahead) and we show the results when 2, 3, 5, and 10 steps look-ahead are used to compute the loss function. Each problem is solved 5 times with different random initialisations of 5 data points. The number of allows evaluations is 10 times the dimensionality of the problem. This allows us to compare the average performance of each method on each problem. As baseline we use the myopic expected loss, el. For comparative purposes we used other two loss functions are commonly used in the literature. In particular, we use the Maximum Probability of Improvement, mpi, and the gp lower confidence bound, gp-lcb. In this later case we set the parameter that balances exploration and exploitation to 1. See (Snoek et al., 2012) for details on these loss functions. All acquisition functions are optimised using the dividing rectangles algorithm direct (Jones et al., 1993). As surrogate model for the functions we used a gp with a squared exponential kernel plus a bias kernel (Rasmussen and Williams, 2005). The hyper-parameters of the model were optimised by the standard method of maximising the marginal likelihood, using l-bfgs (Nocedal, 1980) for 1,000 iterations and selected the best of 5 random restarts. To compare the methods we used the ‘gap’ measure of performance (Huang et al., 2006), which is defined as
where represents the evaluation of the objective function, is the global minimum, and and are the first and best evaluated point, respectively. To avoid gap measures larger that one due to the noise in the data, the measures for each experiment are normalized across all methods. The initial point was chosen to be the best of the original points used to initialise the models.
Table 2 shows the comparative across different functions and methods. None of the methods used is universally the best but a non myopic loss is the best in 6 of the 11 cases. In 3 cases the full glasses approach is the best of all methods. Specially interesting is the case of the McCormick and the Powers function. In these two cases, to increase the number of steps ahead used to compute the loss consistently improve the obtained results. Note as well that when the glasses algorithm is not the best global method it typically performs closely to the best alternative which makes it a good ‘default’ choice if the function to optimise is expensive to evaluate.
In this paper we have explored the myopia in Bayesian optimisation methods. For the first time in the literature, we have proposed an non-myopic loss that allows taking into account dozens of future evaluations before making the decision of where to sample the objective function. The key idea is to jointly model all future evaluations of the algorithm with a probability distribution and to compute the expected loss by marginalising them out. Because this is an expensive step, we avoid it by proposing a fixed prediction of the future steps. Although this doesn’t make use of the epistemic uncertainty on the steps ahead, it drastically reduces the computation burden of approximating the loss. We made use of the connection of the multiple steps ahead problem with some methods proposed in the batch Bayesian optimisation to solve this issue. The final computation of the loss for each point in the domain is carried out by adapting ep to our context. As previously suggested in Osborne et al. (2009), our results confirm that using a non-myopic loss helps, in practice, to solve global optimisation problems. Interestingly, and as happens with any comparison of loss functions across many objective functions, there is not a universal best method. However, in cases in which glasses is not superior, it performs very closely to the myopic loss, which makes it an interesting default choice in most scenarios.
Some interesting challenges will be addressed in the future such as making the optimisation of the loss more efficient (for which direct is employed in this work): although the smoothness of the loss is guaranteed if the steps ahead are consistently predicted for points close in the space, if the optimization of the steps ahead fails, the optimization of the loss may be challenging. Also, the use of non-stationary kernels, extensions to deal with to high dimensional problems and finding efficient was of sampling many steps ahead will also be analysed.
- Affandi et al.  Raja H. Affandi, Emily .B. Fox, and Ben Taskar. Approximate inference in continuous determinantal processes. In Neural Information Processing Systems 26. MIT Press, 2014.
- Azimi et al.  Javad Azimi, Ali Jalali, and Xiaoli Zhang Fern. Hybrid batch Bayesian optimization. In Proceedings of the 29th International Conference on Machine Learning, 2012.
- Brochu et al.  Eric Brochu, Vlad M. Cora, and Nando De Freitas. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprint arXiv:1012.2599, 2010.
- Cunningham et al.  John P. Cunningham, Philipp Hennig, and Simon Lacoste-Julien. Gaussian probabilities and expectation propagation. arXiv:1111.6832 [stat], Nov 2011. arXiv: 1111.6832.
- Garnett et al.  Roman. Garnett, Michael. A. Osborne, and Stephan. J. Roberts. Bayesian optimization for sensor set selection, page 209–219. ACM, 2010. ISBN 1605589888. doi: 10.1145/1791212.1791238.
- Gillenwater et al.  Jennifer Gillenwater, Alex Kulesza, and Ben Taskar. Near-optimal MAP inference for determinantal point processes. In F. Pereira, C.J.C. Burges, L. Bottou, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 25, pages 2735–2743. Curran Associates, Inc., 2012.
- Ginsbourger et al.  David Ginsbourger, Rodolphe Le Riche, and Laurent Carraro. A multi-points criterion for deterministic parallel global optimization based on Gaussian processes. HAL: hal-00260579, 2009.
- González et al.  Javier González, Joseph Longworth, David James, and Neil Lawrence. Bayesian optimisation for synthetic gene design. NIPS Workshop on Bayesian Optimization in Academia and Industry, 2014.
- González et al.  Javier González, Zhenwen Dai, Philipp Hennig, and Neil D Lawrence. Batch Bayesian optimization via local penalization. arXiv preprint arXiv:1505.08052, 2015.
- Huang et al.  D. Huang, T. T. Allen, W. I. Notz, and N. Zeng. Global optimization of stochastic black-box systems via sequential kriging meta-models. J. of Global Optimization, 34(3):441–466, March 2006. ISSN 0925-5001.
- Jones et al.  D. R. Jones, C. D. Perttunen, and B. E. Stuckman. Lipschitzian optimization without the Lipschitz constant. J. Optim. Theory Appl., 79(1):157–181, October 1993. ISSN 0022-3239.
- Jones  Donald R. Jones. A taxonomy of global optimization methods based on response surfaces. Journal of global optimization, 21(4):345–383, 2001.
- Kulesza and Taskar  Alex Kulesza and Ben Taskar. k-dpps: Fixed-size determinantal point processes. In Lise Getoor and Scheffe, editors, ICML, 2011.
- Kulesza and Taskar  Alex Kulesza and Ben Taskar. Determinantal point processes for machine learning. Foundations and Trends® in Machine Learning, 5(2–3):123–286, 2012. ISSN 1935-8237. doi: 10.1561/2200000044.
- Lizotte  Daniel James Lizotte. Practical Bayesian Optimization. PhD thesis, University of Alberta, 2008. AAINR46365.
Marchant et al. 
Roman Marchant, Fabio Ramos, and Scott Sanner.
Sequential Bayesian optimisation for spatial-temporal monitoring.
Proceedings of the International Conference on Uncertainty in Artificial Intelligence, 2014.
- Martinez-Cantin et al.  Ruben Martinez-Cantin, Nando de Freitas, Eric Brochu, José Castellanos, and Arnaud Doucet. A Bayesian exploration-exploitation approach for optimal online sensing and planning with a visually guided mobile robot. Autonomous Robots, 27(2):93–103, August 2009. ISSN 0929-5593, 1573-7527. doi: 10.1007/s10514-009-9130-2.
Thomas P. Minka.
Expectation propagation for approximate Bayesian inference.In Proceedings of the 17th Conference in Uncertainty in Artificial Intelligence, UAI ’01, pages 362–369, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1-55860-800-1.
- Molga and Smutnicki  Marcin Molga and Czesław Smutnicki. Test functions for optimization needs. NIPS Workshop on Bayesian Optimization in Academia and Industry, 1995. URL www.zsd.ict.pwr.wroc.pl/files/docs/functions.pdf.
- Nocedal  Jorge Nocedal. Updating quasi-Newton matrices with limited storage. Mathematics of Computation, 35(151):773–782, 1980.
- Osborne  Michael Osborne. Bayesian Gaussian Processes for Sequential Prediction, Optimisation and Quadrature. PhD thesis, University of Oxford, 2010.
- Osborne et al.  Michael A. Osborne, Roman Garnett, and Stephen J. Roberts. Gaussian processes for global optimization. In 3rd international conference on learning and intelligent optimization (LION3), pages 1–15, 2009.
- Rasmussen and Williams  Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press, 2005.
- Snoek et al.  Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical Bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems, page 2951–2959, 2012.
- Streltsov and Vakili  Simon Streltsov and Pirooz Vakili. A non-myopic utility function for statistical global optimization algorithms. J. Global Optim., 14(3):283–298, 1999.
S1 Oracle Multiple Steps look-ahead Expected Loss
Denote by the value of the best visited location when looking at evaluations in the future. Note that reduces to the current best lost in the one step-ahead case. It is straightforward to see that
It holds hat
where the integrals with respect to are , since we don’t need to optimize for any location and . Notice that
S2 Formulation of the Oracle Multiple Steps loook-ahead Expected Loss to be computed using Expectation Propagation
Assume that . Then we have that
The first term can be written as follows:
where . We can do this because the regions are disjoint and it holds that . Also, note that the can be replaced within the integrals since within each it holds that . Rewriting the integral in terms of indicator functions we have that
where if and otherwise.
s2.1 Synthetic functions
In this section we include the formulation of the objective functions used in the experiments that are not available in the references provided.