# Optimization for Gaussian Processes via Chaining

In this paper, we consider the problem of stochastic optimization under a bandit feedback model. We generalize the GP-UCB algorithm [Srinivas and al., 2012] to arbitrary kernels and search spaces. To do so, we use a notion of localized chaining to control the supremum of a Gaussian process, and provide a novel optimization scheme based on the computation of covering numbers. The theoretical bounds we obtain on the cumulative regret are more generic and present the same convergence rates as the GP-UCB algorithm. Finally, the algorithm is shown to be empirically more efficient than its natural competitors on simple and complex input spaces.

## Authors

• 4 publications
• 3 publications
• 27 publications
• ### Gaussian Process Optimization with Mutual Information

In this paper, we analyze a generic algorithm scheme for sequential glob...
11/19/2013 ∙ by Emile Contal, et al. ∙ 0

• ### Parallelizing Exploration-Exploitation Tradeoffs with Gaussian Process Bandit Optimization

Can one parallelize complex exploration exploitation tradeoffs? As an ex...
06/27/2012 ∙ by Thomas Desautels, et al. ∙ 0

• ### Stochastic Process Bandits: Upper Confidence Bounds Algorithms via Generic Chaining

The paper considers the problem of global optimization in the setup of s...
02/16/2016 ∙ by Emile Contal, et al. ∙ 0

• ### BdryGP: a new Gaussian process model for incorporating boundary information

Gaussian processes (GPs) are widely used as surrogate models for emulati...
08/23/2019 ∙ by Liang Ding, et al. ∙ 0

• ### Harnessing Heterogeneity: Learning from Decomposed Feedback in Bayesian Modeling

There is significant interest in learning and optimizing a complex syste...
07/07/2021 ∙ by Kai Wang, et al. ∙ 4

• ### Adversarially Robust Optimization with Gaussian Processes

In this paper, we consider the problem of Gaussian process (GP) optimiza...
10/25/2018 ∙ by Ilija Bogunovic, et al. ∙ 0

• ### Gaussian Process Optimization with Adaptive Sketching: Scalable and No Regret

Gaussian processes (GP) are a popular Bayesian approach for the optimiza...
03/13/2019 ∙ by Daniele Calandriello, et al. ∙ 4

##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## 1 Introduction

Optimizing an unknown and noisy function is at the center of many applications in the field of machine learning

[1]. The goal of a sequential optimization procedure may be either seen as maximizing the sum of the outputs (or rewards) received at each iteration, that is to minimize the cumulative regret widely used in bandit problems, or as maximizing the best reward received so far, that is to minimize the simple regret. This task becomes challenging when we only have mild information about the unknown function. To tackle this challenge, a Bayesian approach has been shown to be empirically efficient [2, 3, 4, 5, 6]. In this approach, we model the unknown function as a realization of a Gaussian Process (GP) which allows to control the assumptions we put on the smoothness of the function by choosing different kernels [7]. In order to prove theoretical guarantees on the regret, algorithms in the literature typically rely on high probabilistic upper confidence bounds (UCB) [8, 9, 10, 11, 12]. In these works and many others, the UCB is obtained with a union bound over all the points of a discretization of the input space. The major drawback of this approach is that the UCB depends on the cardinality of the discretization instead of the complexity of the input space itself. As a consequence, the convergence rates derived on the regret become arbitrary large when the discretization becomes finer and finer. Aiming at filling this gap between theory and practice we propose an efficient computation of chaining for Bayesian Optimization. Chaining, that has been recently studied in the context of sequential optimization (see [13] for bandit with known horizon or [14] in the case of online regression), appears to be an ideal tool to capture the complexity of the search space with respect to the smoothness of the function. The contribution of this paper is twofold: we first introduce a novel policy based on the computation of covering numbers called the Chaining-UCB that can be seen as a generalization of the GP-UCB algorithm with automatic calibration of the exploration/exploitation tradeoff for arbitrary kernels and search space. On the other hand, we provide theoretical guarantees on its regret with the same convergence rates as its competitors, without depending on the cardinality of the discretization of the search space. The rest of the paper is organized as follows: in Section 2, we present the framework of our analysis, the basic properties of GP and we introduce the Chaining-UCB algorithm. In Section 3, we present and discuss the upper bound on the regret. Finally in Section 4 we compare the empirical performances 111The MATLAB code is available on http://econtal.perso.math.cnrs.fr/software/ of the Chaining-UCB algorithm to its natural competitors on simple input spaces, that is , and complex input spaces, this is directed graphs space.

## 2 The Chaining-UCB algorithm

#### Bayesian optimization framework.

Let be the unknown function we want to optimize, where is the input space which is not necessarily a subset of . We assume that is a realization of a centered Gaussian process with known kernel satisfying for all . To avoid measurability issues we suppose that is a finite set with arbitrary cardinality (see [15, 16] for more details). A sequential optimization algorithm iterates two steps: it first chooses based on , and next gets the noisy observation where are independent Gaussians

with known variance

. Let be the set of queried points after iterations and

the associated observations packed in vector form. Unlike the work of

[13] this paper considers that the time horizon is unknown. For GP, the distribution of conditioned on is a non-centered GP of mean and kernel computed as follows for all :

 μn+1(x)=kn(x)⊤C−1nYn ~% {}and~{} kn+1(x,x′)=k(x,x′)−kn(x)⊤C−1nkn(x′), (1)

where is the kernel vector between and , and with the kernel matrix [7]. We also define the variance and the pseudo-distance  :

 dn(x,x⋆)=√σ2n(x⋆)−2kn(x,x⋆)+σ2n(x). (2)

Note that . To measure the complexity of we will compute covering numbers with respect to .

#### An upper confidence bound algorithm via chaining.

At the core of our strategy to control the regret of the algorithm we need an upper confidence bound (UCB) on for all . A naive approach uses a union bound on , resulting in a factor in the UCB, which is not appropriate when is a numerical discretization of a continuous space, typically a multi-dimensional grid of high density. We use the chaining trick [17, 18, 19] to get a UCB relying on the covering numbers of with respect to instead of its cardinality. In that way the algorithm adapts to arbitrary input spaces. The main element of our algorithm is the computation of hierarchical -covers of . We say that a subset is an -cover of when for all , , where . The covering numbers are the cardinality of the smallest -cover of for the pseudo-distance , and the function Cover() in Algorithm 1 returns such a set. The Chaining-UCB algorithm then queries the objective function at the point maximizing the UCB obtained by chaining. The computation of an optimal -cover is -hard, but we can easily build an efficient approximation as shown in Algorithm 2 and discussed in Section 4.

## 3 Main results

#### Guarantees on the regret.

In the following theorem we provide a high probabilistic upper bound on the instantaneous regret incurred by Algorithm 1 in terms of the posterior deviations and covering numbers. This inequality is used in the subsequent corollary to obtain upper bounds on its cumulative and simple regrets.

###### Theorem 1.

For any finite , let be the queries of the Chaining-UCB algorithm on sampled from a where . For , using the notations and

, we have with probability at least

that for all :

 supx⋆∈Xf(x⋆)−f(xn)≤σn(cn,δ−6logσn)+9∑i:2−i<σn2−i√logN(X,dn,2−i).

In order to simplify this inequality and get convergence rates for the cumulative regret and simple regret , it is necessary to add some assumptions on and . Corollary 1 gives an example of the rates we obtain for the usual Squared-Exponential kernel and in .

###### Corollary 1.

For the SE kernel and a compact , the Chaining-UCB algorithm returns a sequence such that and .

The proof of Theorem 1 employs the chaining trick to get a local control on the supremum of a non-centered GP. Since this requires to define additional structures, the proofs are postponed to Appendix A. The proof of Corollary 1, in Appendix B, first uses an upper bound on the covering numbers via the Lipschitz property of the SE kernel. It then applies the inequality about the mutual information proven in Theorem 5 of [9] to obtain the given regret rates. It is straightforward to apply this technique to other cases like linear kernels or Matérn kernels with parameter , since both the covering numbers and the mutual information can be bounded by similar techniques.

#### A flexible algorithm.

A reader familiar with classical chaining may ask why we use the bad-looking sum instead of the Dudley integral. Even if the Dudley integral is simple to bound for certain kernel and space , we want Algorithm 1 to be able to adapt to all search space without having to tune its parameters. By computing the successive covering numbers, the Chaining-UCB algorithm calibrates automatically the exploration-exploitation tradeoff. This fact contrasts with previous algorithms like GP-UCB where the input parameter depends either on the cardinality or on the Lipschitz constants of the kernel (see Theorems 1 and 2 of [9]). On the computational side, the discrete sum limits the number of -covers which need to be computed to only the such that . Thanks to their geometrical decay, these numbers remain low in practice. Figure 1 illustrates the exploration/exploitation tradeoff we obtain with this discrete sum on a 1D toy example, compared to the tradeoff computed with a union bound as in GP-UCB. In Figure 1 a constant term is subtracted from the UCB in order to set the minimum of the exploration terms to zero in both Chaining-UCB and GP-UCB.

## 4 Practical considerations and experiments

#### Computing the ϵ-covers efficiently.

As mentioned previously the computation of an optimal -cover is -hard. We demonstrate here how to build in practice a near-optimal -cover using a greedy algorithm on graph. First, remark that for any fixed we can define a graph where the nodes are the elements of and there is an edge between and if and only if . The size of this construction is . The sparse structure of the underlying graph can be exploited to get an efficient representation. The problem of finding an optimal -cover reduces to the problem of finding a minimal dominating set on . We can therefore use the greedy Algorithm 2 which enjoys an approximation factor of , where is the maximum degree of (see for example [20] for a proof of -hardness and approximation results). This construction leads to an additional (almost constant) term of in the right-hand side of Theorem 1. Finally, note that this approximation is optimal unless as shown in [21].

#### Experiments.

In this section we compare the ability of the Chaining-UCB algorithm to find the maximum of an unknown function against the GP-UCB algorithm from [9] and the Random search. For both the Chaining-UCB and the GP-UCB algorithms the value for was set to . The Random approach selects the next query uniformly among the unknown locations. It gives a baseline to grasp the scale of the performances of both algorithms. All three strategies are initialized with a set of noisy observations sampled uniformly over . Figure 2 shows the empirical mean of the simple regret

over 32 runs. In every experiments the standard deviation of the noise was set to

and the search space is discrete with . The SE experiment consists in generating GPs drawn from a two dimensional isotropic SE kernel, with the kernel bandwidth set to . The search space is a uniform design in a square of length . The Himmelblau experiment is a two dimensional polynomial function based on the Himmelblau’s function with the addition of a linear trend. It possesses several local maxima which makes the global optimization challenging. The kernel used by the algorithm is an isotropic SE kernel with the bandwidth chosen by maximizing the marginal likelihood of a subset of training points. Finally we consider the task of optimizing over graphs. Global optimization of graphs can model complex problems as in industrial design, network analysis or computational biology. The input space is the set of directed graphs with less than nodes. The kernel is the shortest-path kernel [22] normalized such that for . Note that in this synthetic assessment we do not address the question faced in practice of choosing the prior. We further mention that the kernel matrix can be efficiently computed by pre-processing all pairs of shortest paths for each graph with Floyd-Warshall’s algorithm. Figure 2 shows the Chaining-UCB algorithm is empirically more efficient than the GP-UCB algorithm on the three test cases. We remark that in practice, unlike GP-UCB, we may use a design with without affecting the performance of Chaining-UCB. However generating a GP costs which limits the tractability of synthetic assessments.

#### Conclusion.

The theorem we derived and the experiment we performed suggest that the automatic calibration of the exploration/exploitation tradeoff by the hierarchical -covers effectively adapts to various settings. This chaining approach is a promising step toward generic, sound and tractable algorithms for Bayesian optimization.

## References

• [1] J. Snoek, H. Larochelle, and R. P. Adams. Practical bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems 25, pages 2960–2968, 2012.
• [2] J. Mockus. Bayesian approach to global optimization. Mathematics and its applications. Kluwer Academic, 1989.
• [3] D. R. Jones, M. Schonlau, and W. J. Welch. Efficient global optimization of expensive black-box functions. Journal of Global Optimization, 13(4):455–492, December 1998.
• [4] M. Osborne. Bayesian Gaussian processes for sequential prediction, optimisation and quadrature. PhD thesis, Oxford University New College, 2010.
• [5] P. Hennig and C. J. Schuler. Entropy search for information-efficient global optimization. Journal of Machine Learning Research, 13:1809–1837, 2012.
• [6] E. Contal, V. Perchet, and N. Vayatis. Gaussian process optimization with mutual information. In Proceedings of the 31st International Conference on Machine Learning (ICML-14), pages 253–261, 2014.
• [7] C. E. Rasmussen and C. Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
• [8] A. Krause and C.S. Ong. Contextual Gaussian process bandit optimization. In Advances in Neural Information Processing Systems 24, pages 2447–2455, 2011.
• [9] N. Srinivas, A. Krause, S. Kakade, and M. Seeger. Information-theoretic regret bounds for Gaussian process optimization in the bandit setting. IEEE Transactions on Information Theory, 58(5):3250–3265, 2012.
• [10] N. de Freitas, A. J. Smola, and M. Zoghi. Exponential regret bounds for Gaussian process bandits with deterministic observations. In Proceedings of the 29th International Conference on Machine Learning. icml.cc / Omnipress, 2012.
• [11] E. Contal, D. Buffoni, A. Robicquet, and N. Vayatis. Parallel Gaussian process optimization with upper confidence bound and pure exploration. In Machine Learning and Knowledge Discovery in Databases (ECML), volume 8188, pages 225–240. Springer Berlin Heidelberg, 2013.
• [12] J. Djolonga, A. Krause, and V. Cevher. High-dimensional gaussian process bandits. In Advances in Neural Information Processing Systems, pages 1025–1033, 2013.
• [13] S. Grunewalder, J-Y. Audibert, M. Opper, and J. Shawe-Taylor. Regret bounds for Gaussian process bandit problems. In

Proceedings of the International Conference on Artificial Intelligence and Statistics

, pages 273–280. MIT Press, 2010.
• [14] P. Gaillard and S. Gerchinovitz. A chaining algorithm for online nonparametric regression. Proceedings of the Conference on Learning Theory (COLT), 2015.
• [15] S. Boucheron, G. Lugosi, and P. Massart. Concentration inequalities: A nonasymptotic theory of independence. Oxford University Press, 2013.
• [16] E. Giné and R. Nickl. Mathematical foundations of infinite-dimensional statistical models, 2015.
• [17] M. Talagrand. Upper and Lower Bounds for Stochastic Processes: Modern Methods and Classical Problems, volume 60. Springer-Verlag Berlin Heidelberg, 2014.
• [18] D. Pollard. Empirical processes: Theory and applications. NSF-CBMS Regional Conference Series in Probability and Statistics, 2:pp. i–iii+v+vii–viii+1–86, 1990.
• [19] R.M. Dudley. The sizes of compact subsets of hilbert space and continuity of gaussian processes. Journal of Functional Analysis, 1(3):290–330, 1967.
• [20] David S Johnson. Approximation algorithms for combinatorial problems. In

Proceedings of the fifth annual ACM symposium on Theory of computing

, pages 38–49. ACM, 1973.
• [21] R. Raz and S. Safra. A sub-constant error-probability low-degree test, and a sub-constant error-probability pcp characterization of np. In Proceedings of the twenty-ninth annual ACM symposium on Theory of computing, pages 475–484. ACM, 1997.
• [22] Karsten M Borgwardt and Hans-Peter Kriegel. Shortest-path kernels on graphs. In Data Mining, Fifth IEEE International Conference on, pages 8–pp. IEEE, 2005.

## Appendix A Proof of Theorem 1

In this section, we first demonstrate how chaining is used to analyze the supremum of a centered GP. Since our aim is to derive an efficient algorithm, we put some effort to keep low values for the constants. The next subsection shows how this simply extends to non-centered GP and how to build the UCB algorithm with the chaining bound. Finally we express the upper bounds in terms of the covering numbers of .

### a.1 Using covering trees to bound the supremum of a GP

Let us fix an iteration and the observations , we define the centered posterior of conditioned on , which is a Gaussian process where is defined in Equation 1. By Cramer-Chernoff’s method [15] we have for all and that:

 Pr[fn(x⋆)−fn(x)>dn(x⋆,x)√2u]

where the pseudo-distance is defined Equation 2. We use classical chaining [17, 19] to obtain concentration inequalities on with respect to the covering numbers. The idea of chaining performs as follows. Let be a sequence of mappings such that and holds for all . We then have the telescopic sum,

 supx⋆∈Xfn(x⋆)−fn(x)=supx⋆∈X∑i≥1fn(πi(x,x⋆))−fn(πi−1(x,x⋆)). (4)

Since may be arbitrary large we cannot perform a union bound over all in Equation 4. Instead we enforce to take its values in a discrete subset of formed with carefully chosen points such that is small, so we can use Equation 3. By controlling we are able to achieve a union bound over and . We describe here the sequence of subsets of , which satisfy , for all and . For all level we fix an exponentially decreasing radius and define as the smallest -cover of containing :

 Ti∈argminT⊇Ti−1dn(x,T)≤ϵi ∀x∈X|T|. (5)

The usual analysis of the chaining upper bound is not suitable for sequential optimization since our aim is to bound for a specific chosen by the algorithm. In this respect, we focus on finer results which keep track on the distance . Unfortunately, this needs to define additional theoretical quantities and pay more attention to the sharpness of the inequalities involved. We endow with a tree structure with respect to and where the root is and the parent of an element of is its closest point among the elements of . Formally the parent relationship satisfies that for all and for the root. We denote by the ancestor of at level , that is the element of such that there exists a branch where for . The sequence is illustrated on Figure 3. The path from to given by works as follows: we start and stay on until one parent of at level is not too far. Precisely for each and level , is either or :

 πi(x,x⋆)={pi(x⋆) if ϵi

Figure 3 highlights the path defined by . Using the definition of and we have , and if then and the -th summand in Equation 4 is equal to zero. Now using the concentration inequality of Equation 3 we get for all :

 Pr[fn(πi(x,x⋆))−fn(πi−1(x,x⋆))>ϵi−1√2ui]

Thanks to the tree structure , . We obtain for all :

 Pr[⋃x⋆∈X⋃i>0{fn(πi(x,x⋆))−fn(πi−1(x,x⋆))>ϵi−1√2ui}]<∑i>0(|Ti|+1)e−ui.

For any , we set , so that . Combining the previous union bound with the chaining property we have a generic upper bound:

 ∀x∈X,Pr⎡⎣supx⋆∈Xfn(x⋆)−fn(x)>supx⋆∈X∑i:ϵi

### a.2 From classical chaining to optimization

The previous inequality provides us with a powerful tool to prove upper bounds on the regret. We use two steps to be able to build the algorithm. We first adapt this result to non-centered processes. We then split the sum in two terms, one for and one for , and we demonstrate that the queries made by Algorithm 1 allow to control both terms.

At iteration , the posterior process of conditioned on has a mean and covariance defined in Equation 1. We fix and we note the covering tree with respect to defined in the previous subsection. We denote the associated summands of Equation 6, where . In order to obtain an analogous result for , it suffices to replace in Equation 4 by the centered process and remark that the terms collide, we get:

 ∀x∈X,Pr⎡⎣supx⋆∈Xf(x⋆)−f(x)−μn(x⋆)+μn(x)>supx⋆∈X∑i:ϵi

With some abuse of notation, we write in the sequel and . We now decompose the indices of the sum in two sets and take the supremum on both:

 supx⋆∈X∑i:ϵi

Since for Algorithm 1 we have , we obtain for all that . We thus prove with probability at least that:

 supx⋆∈Xf(x⋆)−f(xn)≤∑i:ϵi<σnHni+supx⋆∈X∑i:σ⋆n≤ϵiϵi

We simplify this inequality by checking that the second sum on the right side is smaller than twice the first sum. This involves tedious but elementary calculus using the geometric decay of , the linearity of in and the fact that . It suffices to verify that for any , . Without loss of generality we consider that . Let , and . Since and increases as increases, we have on one side , and on the other side . With we get . Plugging this result in Equation 7 we have with probability at least that . If we set to for any , that is , we prove by a union bound over all that with probability at least :

 ∀n>0, supx⋆∈Xf(x⋆)−f(xn)≤3∑i:ϵi<σnϵi−1√2uni. (8)

### a.3 Bounding with covering numbers

In order to relate Equation 8 to the size of the input space we bound the cardinality by the covering numbers of . In fact by construction of we know that , which is less than . Note that the factor is far from being tight for large , but the Chaining-UCB algorithm does not need this crude upper bound since it only uses . Splitting in three factors, the LHS of Equation 8 is smaller than . Using the fact that and that for any , and , we finally get that with probability at least :

 supx⋆∈Xf(x⋆)−f(xn)≤9∑i:ϵi<σnϵi√logN(X,dn,ϵi)+6σn√logn2π436δ+15σn−6σnlogσn,

proving Theorem 1.

## Appendix B Proof of Corollary 1

To prove Corollary 1 we first remark that for the SE kernel the pseudo-distance enjoys for all . By definition of we know that for all . When , it follows that . Plugging this into Theorem 1 we obtain that . We then invoke the inequality proven in Theorem 5 of [9] which shows that . We see by maximizing under this constraint with Lagrange multipliers that . The rate for directly follows from this result.