1 Introduction
It has long been observed that for wellstudied computational problems for which several highperformance algorithms are available, there is typically no single algorithm that dominates all others on all problem instances. Instead, different algorithms perform best on different types of problem instances – a phenomenon also known as performance complementarity, which is often incorrectly attributed to an interesting theoretical result known as the nofreelunch (NFL) theorem (Wolpert and Macready, 1995, 1997). In the context of search problems, the NFL theorem strictly only applies if arbitrary search landscapes are considered, while the instances of basically any search problem of interest have compact descriptions and therefore cannot give rise to arbitrary search landscapes (Culberson, 1998). Performance complementarity has been observed for practically all NPhard decision and optimisation problems; these include propositional satisfiability, constraint satisfaction, a wide range of planning and scheduling problems, mixed integer programming and the travelling salesperson problem, as well as a broad range of continuous optimisation, machine learning and important polynomialtimesolvable problems (e.g., sorting and shortest path finding). For these and many other problems, theoretical results stating which algorithmic strategies work best are restricted to very limited classes of problem instances, so that it is generally unknown a priori which of several algorithms should be used to solve a given instance.
This gives rise to the increasingly prominent perinstance algorithm selection problem: given a computational problem, a set of algorithms for solving this problem, and a specific instance that needs to be solved, determine which of the algorithms can be expected to perform best on that instance. This problem has already been considered in the seminal work by Rice (1976), but it took several decades before practical perinstance algorithm selection methods became available (see, e.g., Cook and Varnell, 1997; LeytonBrown et al., 2003; Xu et al., 2008). Since then, the problem and algorithms for solving it have steadily gained prominence, and by now have given rise to a large body of literature. Indeed, perinstance algorithm selection techniques have produced substantial improvements in the state of the art in solving a large range of prominent computational problems, including propositional satisfiability (SAT) and the travelling salesperson problems (TSP) (Xu et al., 2008, 2012; Kerschke et al., 2017).
It is important to note that there are several concepts that are quite closely related to that of perinstance algorithm selection, notably, perset algorithm selection, algorithm configuration, algorithm schedules and parallel algorithm portfolios, which are all discussed in further detail in Section 2. Unfortunately, there is some potential for confusion, especially between perinstance algorithm selection and parallel algorithm portfolios, since in the literature, the term portfolio is sometimes used to refer to algorithm selectors. Furthermore, some of the most prominent and successful algorithm selection approaches from the literature, such as SATzilla (Xu et al., 2008) and AutoFolio (Lindauer et al., 2015b), implement combinations of algorithm scheduling and perinstance selection. While we will briefly discuss these more complex systems, along with approaches that select more than one algorithm to be run on a given problem instance, the focus of this survey is on pure perinstance algorithm selection, as outlined above and defined formally in Section 2.
We note that perinstance algorithm selection can be applied to optimisation problems, where the goal is to find an optimal (or best possible) solution according to a given objective function, as well as to decision problems, where one wants to determine, as quickly as possible, whether a solution satisfying certain conditions exists. Furthermore, it is useful to distinguish between continuous problems, where the components of a possible solution are real numbers (possibly constrained to a given interval), and discrete problems, where candidate solutions are discrete objects, such as graphs, permutations or vectors of integers.
Several surveys on algorithm selection have been published over the last decade. In the first extensive survey in this area, SmithMiles (2009)
summarised developments in the metalearning, artificial intelligence and operations research communities. Adopting a crossdisciplinary perspective, she combined contributions from these areas under the umbrella of a “(meta)learning” framework, which permitted her to identify parallel and closely related developments within these rather wellseparated communities. However, this survey was published a decade ago and therefore does not cover recent developments and improvements to the state of the art in this fastmoving research area.
A more recent overview on algorithm selection was published by Kotthoff (2014). His survey presents an extensive, valuable guide to the automated algorithm selection literature up to 2014 and provides answers to several important questions, such as (i) what are the differences between static and dynamic portfolios, (ii) what should be selected (single solver, schedule, different candidate portfolios), (iii) what are the differences between online and offline selection, (iv) how should the costs for using algorithm portfolios be considered, (v) which prediction type (classification, regression, etc.) is most promising when training an algorithm selector, and (vi) what are differences between static and dynamic, as well as lowlevel and highlevel features. Unfortunately, Kotthoff’s survey is restricted to algorithm selection for discrete problems and does not cover in any detail problem instance features, which provide the basis for perinstance algorithm selection.
Those two limitations where – at least partially – addressed by Muñoz Acosta et al. (2013). Although the title (“The Algorithm Selection Problem on the Continuous Optimization Domain”) appears to suggest otherwise, their survey mostly addresses the paucity of work on algorithm selection for continuous optimisation problems and the challenges arising in this context. Rather than providing an overview of algorithm selection approaches in this area, Muñoz Acosta et al. (2013) summarise promising results on discrete problems and hint at the possibility of achieving similar results in continuous optimisation. In their followup survey, Muñoz Acosta et al. (2015b) provide further insights into the existing ingredients for algorithm selection in the domain of continuous optimisation: benchmarks, algorithms, performance metrics, and problem characteristics obtained by exploratory landscape analysis. Still, they do not cover any work describing automated algorithm selection in this domain.
Our goal here is to not only update, but also to complement and extend these previous surveys. Firstly, we cover work on algorithm selection for discrete and continuous problems; as a result, we can compare the difficulties, challenges and solutions found in those domains. Secondly, one of the most important ingredients for successful algorithm selection approaches are informative (problemspecific) features. We therefore provide an overview of several promising feature sets and discuss characteristics that have been demonstrated to provide a strong basis for algorithm selection. Thirdly, we discuss several problems closely related to (and sometimes confused with) algorithm selection, such as automated algorithm configuration, algorithm schedules and parallel portfolios, pointing out differences, similarities and synergies. Of course, in light of the considerable and fastgrowing body of literature on and related to algorithm selection, we cannot provide comprehensive coverage; instead, we selected contributions based on their impact, promise and conceptual contributions to the area.
The remainder of this survey article is structured as follows. In Section 2, we formally define the perinstance algorithm selection problem and situate it in the context of related problems, such as automated algorithm configuration. Next, Section 3 provides an overview of instance features for discrete and continuous optimisation problems that provide the basis for automated algorithm selection. Successful applications of algorithm selection in discrete and continuous optimisation are discussed in Sections 4 and 5, respectively. Finally, Section 6 provides additional perspectives on algorithm configuration and outlines several open challenges.
2 Algorithm Selection and Related Problems
We consider the selection of algorithms for a given decision or optimisation problem . Specifically, the perinstance algorithm selection problem can be formulated as follows (see also Rice, 1976): Given a set of instances of a problem , a set of algorithms for and a metric that measures the performance of any algorithm on instance set , construct a selector that maps any problem instance to an algorithm such that the overall performance of on is optimal according to metric .
Of course, in general, we cannot hope to efficiently find perfect solutions to the perinstance algorithm selection problem, and instead, we strive to find selectors whose performance is as close as possible to that of a perfect selector on instance set . This is typically achieved by making use of informative and cheaply computable features of the given problem instance . A general overview on the interplay of instance features, algorithm performance data, and algorithm selection is shown in Figure 1. The features are of key importance, and we will discuss them in more detail in the following section of this article.
We note that the performance of a (hypothetical) perfect perinstance algorithm selector, often also referred to as an oracle selector or virtual best solver (VBS), provides a lower bound on the performance of any realistically achievable algorithm selector (where we assume, w.l.o.g., that the given performance measure is to be minimised), and is often used in the context of assessing selector performance.
Another useful concept is that of the single best solver (SBS), which is the algorithm with the best performance among all the . The SBS is the solution to the closely related perset algorithm selection problem, and its performance provides a natural upper bound on the performance of any reasonable perinstance algorithm selector. Furthermore, the difference or ratio between the performance of the SBS and VBS, also known as the VBSSBS gap, gives an indication of the performance gains that can be realised, in the best case, by perinstance algorithm selection, and the fraction of the VBSSBS gap closed by any perinstance algorithm selector provides a measure of its performance (see, e.g., Lindauer et al., 2017b). Stateoftheart perinstance algorithm selectors for combinatorial problems have demonstrated to close between 25% and 96% of the VBSSBS gap (see, e.g., Lindauer et al., 2015b). It is important to note that the VBSSBS gap is large when the given set of algorithms shows high performance complementarity on instance set , i.e., when different perform best on different , and those algorithms that are best on some instances perform quite poorly on others. Generally, perinstance algorithm selection can be expected to achieve large performance gains over the single best algorithm if there is high performance complementarity within and there is a set of sufficiently cheaply computable and informative instance features that can be leveraged in learning a good mapping from instances to algorithms.
It is very important to distinguish between perset algorithm selection and perinstance algorithm selection. The former does not require any instance features and is typically done by exhaustive evaluation of all given algorithms on a set of problem instances deemed to be representative for those to be solved later. The connection between perinstance algorithm selection and related problems is shown in Figure 2. In many ways, algorithm competitions, such as the international SAT and planning competitions (see, e.g., Järvisalo et al., 2012; Vallati et al., 2015), can be seen as identifying solutions to perset algorithm selection problems for broad sets of interesting instances, and competition winners are often seen as the single best algorithm for the respective problem. Sometimes, to reduce the computational cost for perset algorithm selection, racing methods are used. These run candidate algorithms on an increasing number of instances, eliminating those from consideration whose performance is significantly below that of others, based on a statistical test (see, e.g., Maron and Moore, 1994; Birattari et al., 2002).
Perset algorithm configuration can be seen as a special case of algorithm configuration, a practically very important problem that can be described as follows: Given an algorithm whose performance (but not semantics) is affected by the settings of parameters , a set of possible values for (called configurations of ), a set of problem instances and a performance metric , find a configuration of that achieves optimal performance on according to . Note that the set of configurations can be seen as corresponding to a set of algorithms, of which we wish to select the one that performs best. The key difference to algorithm selection is that this set can be very large, since it arises from combinatorial combinations of values of the individual parameters , which, in some cases, can take continuous values, leading to (potentially) uncountably infinite sets of algorithm configurations over which we have to optimise. Realistic algorithm configuration scenarios typically involve tens to hundreds of parameters (see, e.g., Hutter et al., 2009, 2011; LópezIbáñez et al., 2016; Ansótegui et al., 2015; Thornton et al., 2013; Kotthoff et al., 2017). Therefore, perset algorithm selection techniques are typically not directly applicable to algorithm configuration, although racing techniques can be extended to work well in this case (see, e.g., LópezIbáñez et al., 2016; Pérez Cáceres et al., 2017)
. Perset algorithm configuration is closely related to hyperparameter optimisation in machine learning; the main difference is that in algorithm configuration, performance is to be optimised on a possibly diverse set of problem instances, which often requires trading off performance on some instance against that achieved on others. In a sense, the typical hyperparameter optimisation problem encountered in machine learning is analogous to configuring a parameterised algorithm for performance on a single problem instance.
Since typical procedures used for building perinstance algorithm selectors have design choices that can be exposed as parameters, algorithm configuration techniques can be applied to optimise their performance on specific (sets of) selection scenarios. This has been done, with considerable success, in the recent AutoFolio selection system by Lindauer et al. (2015b), which we will discuss in further detail in Section 4.
The perinstance variant of the algorithm configuration problem, which can be seen as a generalisation of perinstance algorithm selection, largely remains an open challenge (see, e.g., Hutter et al., 2006; Belkhir et al., 2016, 2017), and we briefly discuss it further in Section 6.
Performance complementarity within a set of algorithms can be leveraged in ways that differ from perinstance algorithm configuration. One prominent approach is that of a parallel algorithm portfolio, where each algorithm from a given set is run in parallel on a given problem instance (see, e.g., Huberman et al., 1997; Gomes and Selman, 2001; Fukunaga, 2000). When applied to a decision problem, all runs are terminated as soon as one of the component algorithms has solved the given instance ; for optimisation problems, the best solution achieved by any of the component algorithms at any given time is returned as the solution of the entire portfolio. Parallel algorithm portfolios are conceptually similar to ensemble methods in machine learning (see, e.g., Dietterich, 2000; Rokach, 2010). The key difference is that ensemble methods aggregate the results from the various component algorithms, e.g., by weighted or unweighted averaging.
When run on parallel hardware, algorithm portfolios typically achieve performance very close to that of the VBS in terms of wallclock time, at the price of parallelism of degree equal to the number of algorithms in . Of course, parallel portfolios can be run at lower actual degrees of parallelism, and even fully sequentially, using taskswitching, as provided, e.g., by the operating system; in that case, the wallclock time is typically close to that of
times the performance of the VBS in terms of the time required to solve a given instance of a decision problem, or to achieve a certain solution quality in case of an optimisation problem. Most of this overhead, which for large sets of algorithms can be very substantial, can be avoided by using perinstance algorithm selectors instead of parallel portfolios. Nevertheless, especially in the area of evolutionary computation, the concept of parallel algorithm portfolios has given rise to a growing body of research, in which the basic concept is often combined with additional techniques to achieve improved performance
(see, e.g., Tang et al., 2014; Yuen and Zhang, 2015).Although the term algorithm portfolio is sometimes used in the literature to refer to perinstance algorithm selectors and other techniques that leverage performance complementarity within a set of algorithms, we discourage this broad use as it easily leads to confusion between conceptually very different approaches. This potential confusion is easily avoided by restricting the term algorithm portfolio to parallel algorithm portfolios, consistent with the seminal work by Huberman et al. (1997). Recently, the concepts of perinstance algorithm selection and parallel portfolios have been combined, by selecting, on a perinstance basis, several algorithms to be run in parallel (Lindauer et al., 2015a). While this would not make sense in the context of a perfect algorithm selector, it can limit the impact of the poor selection decisions sometimes made in practice.
Algorithm schedules provide another way of exploiting performance complementarity (see, e.g., Lindauer, 2014; Lindauer et al., 2016). The key idea is to run a sequence of algorithms from a given set , one after the other, each for a given (maximum) time. Those cutoff times can differ between the stages of the schedule, and some algorithms may not be run at all. Static algorithm schedules, i.e., schedules that have been determined in a perset fashion and are applied uniformly to any given problem instance , can be quite effective and are typically much easier to implement than perinstance algorithm selectors (see, e.g., Roussel, 2012). They are also used in stateoftheart algorithm selection systems during a socalled presolving phase, in order to solve easy problem instances quickly and without the need for computing the instance features required for perinstance selection (see, e.g., Xu et al., 2012; Lindauer et al., 2015b). Unless stated explicitly otherwise, we will in the following, when discussing perinstance algorithm selectors, always refer to the pure perinstance algorithm selection problem, as defined above, without presolving schedules and other extensions found in cuttingedge algorithm selection systems.
Finally, the problem of predicting the performance of an algorithm on a given problem instance is closely related to perinstance algorithm selection. If computationally cheap and accurate performance prediction were possible, evidently, we could use performance predictors for our given algorithms and simply select the one predicted to perform best on . In practice, sufficient accuracy can be achieved for many problems using stateoftheart regression techniques from machine learning, at moderate computational cost (Hutter et al., 2014b), and performance predictors form the basis for one of the main approaches to perinstance algorithm selection. At the same time, other approaches, such as costbased classification, exist and also find use in stateoftheart algorithm selection systems, as explained in the following sections.
3 Features for Discrete and Continuous Problems
Linking algorithm performance on an instance to instance characteristics forms a central part of automated algorithm selection and several related problems. For this purpose, automatically computable features are required, ideally with the following properties: Firstly, features should be informative, in that they allow for a sufficient distinction between different instances; they should also be interpretable, so that feature values enable an expert to gain maximum insight into instance properties. Furthermore, features should be cheaply computable, so that the advantages gained by selecting an algorithm based on them is not outweighed by the cost of feature computation. Features should also be generally applicable, i.e., they should be effectively and efficiently computable for a broad range of problem instances, rather than being restricted, e.g., to small instance sizes. Finally, the features should be complementary, in that redundant sets of features are not only computationally wasteful, but can also cause problems when used by certain machine learning algorithms as a basis for algorithm selection and related problems.
In the following, we provide an overview of commonly used instance features for several prominent discrete and continuous problems – not only to illustrate what kind of features are useful in the context of perinstance algorithm selection, but also to draw attention to an important and somewhat underrated research topic of significant importance to tasks beyond algorithm selection. In particular, informative instance features can provide important insights into strengths and weaknesses of a given algorithm, and hence play a crucial role in devising improvements. We generally distinguish between problemspecific features that are closely based on particular aspects of the problem to be solved, such as the number of clauses in instances of propositional satisfiability problems, and generic features that are more broadly applicable, such as highlevel statistics over information gleaned from short ’probing’ runs of a solver for the given problem.
3.1 Discrete problems
To give concrete examples, and in light of the importance of problemspecific features, we will focus on three of the most prominent and wellstudied discrete combinatorial problems: propositional satisfiability (and related problems), AI planning and the travelling salesperson problem (TSP).
Propositional satisfiability and related problems.
The propositional satisfiability problem (SAT) is to determine whether for a given formula in propositional logic, containing Boolean variables , there exists an assignment of logical values to the variables such that evaluates to ; such a variable assignment is said to satisfy . Typically, the problem is restricted to formulae in conjunctive normal form (CNF), i.e., consists of conjunctions () of socalled clauses, which are disjunctions () of Boolean variables and their negations . A CNFformula evaluates to , if each of its clauses is satisfied simultaneously. SAT is one of the most prominent and intensely studied combinatorial decision problems and has important applications in hard and software verification (see, e.g., Biere et al., 2009). Given the ties to other combinatorial problems, improvements in SAT often also impact widely studied related problems, such as maximum satisfiability (MaxSAT) problem, in which the objective is to find a variable assignment that maximises the number of satisfied CNF clauses.
The first large collection of features for SAT (and thus also MaxSAT) instances was provided by Nudelman et al. (2004b). Despite the rather simple structure of SAT instances, the authors devised nine different feature sets and a total of 91 features, which characterise a given CNF formula from a multitude of perspectives. Eleven problem size features describe SAT instances based on summary statistics of their numbers of clauses and variables. A set of variableclause graph (VCG) features comprises ten node degree statistics based on a bipartite graph over the variables and clauses of a given instance. Interactions between the variables are captured by four variable graph (VG) features; these are the minimum, maximum, mean and coefficient of variation of the node degrees for a graph of variables, in which edges connect pairs of variables that jointly occur in at least one clause. Similarly, the set of clause graph (CG) features contains seven node degree statistics of a graph whose edges connect clauses that have at least one variable in common, as well as three features based on weighted clustering coefficients for the clause graph. Thirteen balance features capture the balance between negated and unnegated variables per clause, their overall occurrences across all clauses, as well as fractions of unary, binary and ternary clauses, whereas six further features quantify the degree to which the given
resembles a Horn formula (a restricted type of CNF formula, for which SAT can be decided efficiently). The solution of a linear program representing the given SAT instance provides the basis for six
LPbased features. Finally, there are two sets of socalled probing features, which are based on performance statistics over short runs of several wellknown SAT algorithms (based on DPLL and stochastic local search, two prominent approaches to solving SAT) and capture the degree to which these make early progress on the given instance.Some of the feature sets – specifically, the CG, VG and LPbased features, as well as some of the VCG, balance and DPLLprobing features – are computationally quite expensive (see, e.g., Xu et al., 2008; Hutter et al., 2014b) and consequently not always useful in the context of practical algorithm selection approaches. Similarly, the algorithm runs for probing features are limited to a very small part of the overall time budget for solving a given instance, to make sure that sufficient time remains available for running the selected SAT solver.
A decade later, Hutter et al. (2014b) – building on the work by Nudelman et al. (2004b) – introduced a set of 138 SAT features. While they removed some features from the earlier sets, much of the set remained the same. The most significant changes were an extension of the CG and VG feature sets by five new features each, as well as three new feature sets accounting for an additional 48 features. The VG feature set was extended by socalled diameter features, which capture statistics based on the set of longest shortest paths from one variable to any other one in the graph. Also, instead of the weighted clustering coefficients based on the CG (as done by Nudelman et al., 2004b), Hutter et al. (2014b) used a set of clustering coefficients that measure the CG’s “local cliqueness”. Furthermore, they introduced 18 novel clause learning features, which summarise information gathered during short runs of a prominent SAT solver, zchaff_rand, that learns conflict clauses during its search for a satisfying assignment (Mahajan et al., 2004)
. Another 18 features are derived from estimates of variable bias obtained from the SAT solver
VARSAT (Hsu and McIlraith, 2009); these features essentially capture statistics over estimates for the probability for variables to be
, or in every satisfying assignment. Finally, Hutter et al. (2014b) proposed to use the actual feature costs, in terms of the running time required for computing each of the 11 feature sets; they noted that the diameter and survey propagation features tend to be expensive to compute and may thus be of limited usefulness in the context of perinstance algorithm selection.A wellknown generalisation of SAT is the problem of answer set programming (ASP; see, e.g., Baral, 2003)
, which deals with determining socalled “answer sets”, i.e., stable models for logic programs. Many combinatorial problems can be presented in ASP in a rather straightforward way and solved, at least in principle, using generalpurpose ASP solvers. Because of the close relationship between ASP and SAT, many features for ASP instances are closely related to the SAT features outlined above. One of the most widely used collection of ASP features has been proposed by
Maratea et al. (2012); it is comprised of 52 features, which can be grouped into four sets. Three of these feature sets closely correspond to wellknown SAT features (Nudelman et al., 2004b) and contain eight problem size, three balance and two proximity to Horn features. In addition, Maratea et al. (2012) proposed 39 ASPspecific features, such as the numbers of true and disjunctive facts, the fraction of normal rules and constraints, and several combinations of the latter.AI planning.
Automated planning (also known as AI planning) is one of the most prominent challenges in artificial intelligence (see, e.g., Ghallab et al., 2004). While there are many variants of AI planning problems, the basic setting (also known as classical planning) involves a set of actions with associated preconditions, deterministic effects and sometimes costs, an initial state and one or more goal states. The objective in satisficing planning is to find a valid plan, i.e., a sequence or partially ordered set of actions that, when applied to the initial state, reach a goal state, or to determine that no valid plan exists. In the optimisation variants of planning problems, the objective is to find plans of minimal length or cost. Most variants of AI planning are at least NPhard, and satisficing classical planning is known to be PSPACEcomplete. AI planning algorithms have important applications, e.g., in robotics, gaming, logistics and software test case generation; they are also used for the operation and management of traffic, energy grids and fleets of shared vehicles.
In classical planning, there is an important distinction between a problem instance and a socalled planning domain. This distinction arises from the fact that states and actions are specified in an abstract way, using socalled predicates and operators that can be instantiated to yield specific properties of states and specific actions, respectively (see, e.g., Ghallab et al., 2004). For example, in a planning problem that involves moving goods using a fleet of trucks, there might be a predicate stating that a specific truck is in a given location, and an operator that moves the truck from one location to another. A planning domain is a class of a planning instances with the same set of specific predicates and operators. Planning domains and instances can be concisely described in a widely used, uniform language called PDDL (Planning Domain Definition Language; see, e.g., Gerevini and Long, 2005).
Howe et al. (1999) were among the first to characterise AI planning instances by simple features, namely, the number of actions, predicates, objects, goals, as well as the number of predicates used to specify the initial state. A decade later, Roberts et al. (2008) introduced a substantially extended set of 41 features, which includes summary statistics of the domain and instance files (16 and three features, respectively), but also captures 13 highlevel features of the given planning instance in terms of its PDDL requirements. They also considered nine features based on the socalled causal graph (CG) (i.e., a graph capturing causal dependencies between states), such as the number of vertices in the CG and their average degree, as well as various metrics computed from the edges of the CG.
Also based on the idea of using graph properties, Cenamor et al. (2013) proposed a total of 47 features, which capture the information contained in causal and domain transition graphs. The latter represent the permissible transitions between states. The causal graph features of Cenamor et al. (2013) can be categorised into four different sets: (i) four general graph properties, (ii) four features based on various ratios of graph properties, (iii) 12 statistical aggregations over the entire graph, and (iv) six additional, highlevel statistics for states with defined values in the goal specifications. The remaining 21 domain transition graph features include (i) three general graph properties (number of edges and states, sum of edge weights) and (ii) 18 statistical features similar to those of the causal graph.
The most recent and extensive collection of AI planning instance features was prepared by Fawcett et al. (2014). It contains 12 sets with 311 features in total, covering most of the features from earlier work, as well as a broad range of new ones. The first three sets extend the 16 domain, three problem and 13 language requirement features from Roberts et al. (2008) by two, four and 11 new features, respectively. Four further feature sets are based on a translation of the given PDDL instance into a finite domain representation (FDR), by means of a wellknown AI planning system, Fast Downward (Helmert, 2006). This FDR representation, as well as information collected during the translation and preprocessing, gives rise to sets of 19, 19 and eight features, respectively. Building on the work of Cenamor et al. (2013), Fawcett et al. (2014) also provide a set of 41 causal and domain transition graph features. Six features are computed from information gathered during the preprocessing phases of LPGtd (Gerevini et al., 2003), another wellknown planning system, while 10 further features capture information produced by the Torchlight local search analysis tool (Hoffmann, 2011); another 16 features are determined based on the trajectories of onesecond probing runs of Fast Downward. Furthermore, the 115 SAT features from Xu et al. (2012) are included, based on a SAT representation of the given planning instance (in form of a CNF with a planning horizon of 10). The final feature set introduced by Fawcett et al. (2014) contains information on whether the previously outlined sets were computed successfully and additionally captures the respective computation times.
Travelling salesperson problem.
The travelling salesperson problem (TSP) is one of the most intensely studied combinatorial optimisation problems. For decades, it has been the subject of a large body of work and continues to be highly relevant for theoretical analyses, design of algorithms and practical applications ranging from logistics to manufacturing (see, e.g., Applegate et al., 2007). In the TSP, given an edgeweighted graph, whose vertices are often called cities and whose edges represent the cost of travelling from one city to another, the objective is to find a Hamiltonian cycle with minimum total weight, i.e., a minimumcost trip that passes through every city exactly once. Most work on the TSP focusses on the special case of the twodimensional Euclidean TSP, where cities are locations in the Euclidean plane, and costs correspond to the Euclidean distances between cities.
The development of features for TSP instances has been initiated by SmithMiles and van Hemert (2011)
, who proposed the following features for characterising a given TSP instance: (1) coordinates of the instance’s centroid, (2) average distance from all cities to the centroid, (3 & 4) standard deviation of distances, as well as fraction of distinct distances within distance matrix, (5) size of the rectangle enclosing the instance’s cities, (6 & 7) standard deviation, as well as coefficient of variation of the normalised nearest neighbour distances, (8) ratio of number of clusters found by GDBSCAN
(Sander et al., 1998)to the number of all cities, (9) variance of number of cities per cluster, (10) ratio between number of outliers and all cities, and (11) the number of cities. Furthermore,
Kanda et al. (2011) and Kovárik and Málek (2012) proposed features derived from the distance matrix of a given TSP instance.Nearly all features from these earlier studies were combined by Mersmann et al. (2013) and further extended, leading to a collection of six TSP feature sets with a total of 68 features, many of which are derived from the distance matrix and from the spatial distribution of the cities in the Euclidean plane. More precisely, the distance matrix is condensed into distance and mode features, and the distribution of cities is captured by a set of cluster features, based on multiple runs of GDBSCAN, as well as convex hull features, which quantify the spread of the cities in the Euclidean plane. The closeness of neighbouring cities is measured by various nearest neighbour statistics, and a final set of features is comprised of the depth and edge costs of the minimum spanning tree for the given TSP instance.
Hutter et al. (2014b) developed a set of 64 TSP features that includes some of the previously outlined instance characteristics as well as new probing features. The latter are based on 20 short runs of a wellknown local search solver (LK; Lin and Kernighan, 1973), as well as single short runs of the stateoftheart exact TSP solver, Concorde (Applegate et al., 2007). Probing features were also used by Kotthoff et al. (2015).
The most comprehensive collection of TSP features was provided by Pihera and Musliu (2014). Their set of 287 features builds on the earlier work by Hutter et al. (2014b), but additionally includes instance characteristics derived from the distances of the cities to the convex hull, the number of intersections of locally optimal tours, and statistics of disjoint tour segments. The largest group of new features is based on strongly and weakly connected components of socalled nearest neighbour graphs, for many values of .
Finally, we see significant potential for new features based on recent work on funnelstructures in the search landscapes associated with TSP instances (Ochoa et al., 2015; Ochoa and Veerapen, 2016). Considering that highly related aspects of global search space structure have shown to play an important role in algorithm selection for continuous optimisation problems (Bischl et al., 2012; Kerschke and Trautmann, 2018), features of this nature may also prove to be useful for discrete optimisation problems, such as the TSP.
Other combinatorial problems.
There is a sizeable body of work on instance characteristics for other combinatorial problems, including the epistasis measures by Davidor (1991) and Fonlupt et al. (1998), indicators for the hardness of quadratic assignment (QAP, Angel and Zissimopoulos, 2002) or constraint satisfaction problems (CSP, Boukeas et al., 2004), features for socalled orienteering problems, which generalise the TSP by distinguishing between static and dynamic locations (Bossek et al., 2018), and variable interaction measures for combinatorial optimisation problems (Seo and Moon, 2007). A detailed discussion of these problems and the respective instance features (some of which are quite generic and can be applied to a range of discrete combinatorial problems) is beyond the scope of this survey; however, we note that these features, like the problemspecific features described earlier in this section, provide a good basis for perinstance algorithm selection and related tasks.
3.2 Continuous problems
We now turn our attention to the optimisation of continuous fitness landscapes (Wright, 1932; Kauffman, 1993). In contrast to discrete optimisation problems, which differ very substantially from each other (consider, for example, SAT vs. TSP) and require problemspecific features for characterising instances, the general idea of continuous optimisation problems can be expressed uniformly, in a rather straightforward way: (w.l.o.g.) find the global minimum of an objective or fitness function , which maps vectors of variables, , from a dimensional decision space (whose values may be subject to additional constraints) to dimensional objective or fitness values (Jones, 1995; Stadler, 2002). Unfortunately, in most realworld scenarios, the exact mathematical representation of the fitness function is unknown. Thus, its optimisation often has to be handled as a blackbox problem, and consequently becomes difficult and expensive. In light of this, it is especially useful to characterise a specific problem by means of (informative) features, based on which it is possible to select a suitable optimisation algorithm.
As there only exist preliminary studies on the characterisation of multiobjective () problems (see, e.g., Kerschke et al., 2016b) – which we will discuss later – we will in the following mainly focus on the manifold of characterisation approaches for singleobjective () continuous optimisation problems.
Singleobjective continuous problems.
Overviews on the early works related to this problem class can be found in Pitzer and Affenzeller (2012), Malan and Engelbrecht (2013), Sun et al. (2014) and Muñoz Acosta et al. (2015b). However, in contrast to recent studies, the majority of the studies covered by those surveys proposed measures for characterising whitebox problems – i.e., problems, whose landscapes are entirely known upfront – rather than cheap, informative and automatically computable landscape features as needed for blackbox problems. Obviously, only the latter are beneficial for automated algorithm selection (or related problems). Nevertheless, we briefly discuss some noteworthy contributions to the characterisation of whitebox problems, as they form the basis for recent developments.
In the 1990s, landscapes were classified into easy and hard problems – from an optimisation algorithm’s point of view. While
Jones and Forrest (1995) (and later Müller and Sbalzarini, 2011) proposed fitness distance correlation as a key characteristic, Rosé et al. (1996) suggested the density of states for solving the binary classification task. Furthermore, socalled epistasis measures (Naudts et al., 1997; Rochet et al., 1997) quantify the influence of single variables (or bits, in case of a bitrepresentation of ) on the problem’s fitness, which in turn can be used to rank the landscapes according to their difficulty. The information content measures from Vassilev et al. (2000) provide another basis for quantifying hardness.In the following decade, attention shifted from characterising problem difficulty to ruggedness. Depending on fitness evolvability portraits (Smith et al., 2002), autocorrelation coefficients, number and distribution of optima (Brooks and Durfee, 2003), or entropy (Malan and Engelbrecht, 2009), landscapes were categorised into rugged, neutral and smooth problems. During that time, researchers also focused on problem multimodality (Preuss, 2015), i.e., the analysis of the problems’ landscapes w.r.t. multiple local and/or global optima. For instance, the barrier trees of Flamm et al. (2002) provided means for identifying basins of attractions, local optima and saddle points within the landscapes, and the dispersion metrics from Lunacek and Whitley (2006) enabled an approximate estimation of the degree of multimodality of a given landscape.
Pitzer and Affenzeller (2012) combined these approaches under the term fitness landscape analysis (FLA). However, as the majority of those characteristics was proposed for whitebox settings, they are not useful for automated algorithm selection. With the introduction of Exploratory Landscape Analysis (ELA), Mersmann et al. (2011) were the first to explicitly develop landscape features for use in blackbox optimisation (BBO) – and hence for algorithm selection in continuous optimisation. As shown in Figure 3, they introduced six feature sets (curvature, convexity, levelset, local search, meta models and distribution), combining a total of 50 features, and used these to predict eight different problem characteristics, such as the global structure (none, weak, strong, deceptive) and the degree of multimodality (none, low, mediocre, high). We note that the attributes of the latter “highlevel” properties can only be assigned by someone with knowledge of the entire problem, whereas the former “lowlevel” features can be automatically computed based on a small, but representative sample of points – the socalled initial design.
Following the idea of automatically computable numerical features, some of the earlier whitebox landscape characteristics have been adapted to the blackbox context. For instance, Muñoz Acosta et al. (2012, 2015a) refined the information content features and extended them by basin of attraction features. Similarly, Abell et al. (2013) proposed hill climbing and random point features, whose general idea stems from the local search and fitness distance correlation methods, respectively, and enhanced them by problem definition measures.
In addition, several new feature sets have been introduced in recent years: Kerschke et al. (2014)
discretised the continuous search space into a grid of cells and used a Markovchaininspired cellmapping approach to obtain features, which measure – amongst others – the sizes of the basins of attraction. However, due to the curse of dimensionality, those features are only practically applicable to lowdimensional problems. A much more scalable measure for the basins of attraction and the global structure of a landscape are the
nearest better clustering features (Kerschke et al., 2015), which provide the means to distinguish funnelshaped from rather random global structures. The most recently published features based on aggregated information of neighbouring points are the bag of local landscape features by Shirakawa and Nagao (2016). Furthermore, the length scale features of Morgan and Gallagher (2015) measure the variable scaling based on the ratio between the change in objective space and distance in decision space for pairs of distinct observations from the initial design.We note that work in this area is not restricted to the development of features or problem characteristics. For instance, Malan et al. (2015) and Bagheri et al. (2017) investigated constrained optimisation problems, whereas Kerschke et al. (2016a) showed that landscape features already possess sufficient information if they are computed based on rather small samples of observations (where is the dimensionality of the given optimisation problem).
Until recently, the simultaneous use of feature sets from different research groups has been cumbersome and hence rarely practiced. However, with the development of flacco (Kerschke, 2017b, c), an Rpackage that provides source code for most of the previously listed ELA features, this obstacle has been overcome. Since then, the complementarity of the various features and their potential usefulness as a basis for algorithm selection has been demonstrated in several studies. We provide a detailed overview of this work later (see Section 5). Note that by using a platformindependent webapplication of the flacco package^{1}^{1}1https://flacco.shinyapps.io/flacco/ (Hanster and Kerschke, 2017), researchers and practitioners, who are unfamiliar with R, can also benefit from this extensive collection of more than 300 landscape features. Belkhir et al. (2016, 2017) were among the first to leverage the ELA features provided by flacco for perinstance algorithm configuration.
Multiobjective continuous problems.
While the characterisation of singleobjective continuous optimisation problems has been studied for over two decades, only preliminary studies have been conducted w.r.t. informative features of multiobjective problems. For instance, Kerschke and Trautmann (2016) used features that have been developed for singleobjective problems and used them to cluster some wellknown and frequently used multiobjective benchmark problems. However, those features do not capture characteristics that are especially important to multiobjective problems, such as interaction effects between the objectives. Eventually, techniques that were originally aimed at measuring variable interactions (see, e.g., Reshef et al., 2011; Sun et al., 2017) could help to overcome these limitations.
Kerschke et al. (2016b, 2018b) investigated locally efficient sets and the corresponding locally optimal fronts (i.e., the multiobjective equivalents of local optima) and proposed measures – on the basis of those sets and fronts – which enable the distinction of multiobjective problems according to their degree of multimodality. Interestingly, those studies revealed that even in the case of rather simple multiobjective problems, strong interaction effects between the objectives exist. Thus, in order to improve the understanding of multiobjective problems, Kerschke and Grimme (2017) introduced new techniques for visualising them. Their approach, dubbed gradient field heatmaps (see, e.g., Figure 4), is based on visualising the decision space, but nevertheless clearly reveals interactions between the objectives – in the form of rugged landscapes with several basins of attractions. Unexpected findings of this kind (see, e.g., Grimme et al., 2018) indicate that research in this area is still at an early stage, leaving many open challenges for future work (see Section 6 for further details).
4 Algorithm Selection for Discrete Problems
Over the years, algorithm selection techniques have achieved remarkable results in several research areas – especially for discrete combinatorial problems (see, e.g., SmithMiles, 2009; Kotthoff, 2014). However, due to the significant differences between various problems, not only the respective instance features, but also solvers and algorithm selectors vary considerably. Therefore, it is impossible to cover all work in this area; instead, in the following, we focus on a small number of particularly wellknown problems: propositional satisfiability (SAT) and related problems, AI planning and the travelling salesperson problem (TSP).
Propositional satisfiability and related problems.
Historically, some of the first and most widely known successes of perinstance algorithm selection have been achieved in the context of solving the propositional satisfiability problem (SAT). We note that several approaches described in the following can be applied to closely related problems (such as MaxSAT) in a rather straightforward way.
SATzilla2003 and SATzilla2007.
Within the highly contested area of SAT, the first AS system to outperform standalone SAT solvers was SATzilla. While its first version, denoted SATzilla2003 (Nudelman et al., 2004a), still showed (minor) weaknesses – it “only” ranked second (twice) and third in the 2003 SAT Competitions – its (enhanced) successor, SATzilla2007 (Xu et al., 2008), won multiple prizes (3x first, 2x second and 2x third) in the 2007 SAT Competition. Despite the differences w.r.t. their success, the general working principles underlying both systems are quite similar. During the actual construction phase, pre and backup solvers are identified, based on the performances of the given solvers on training data. All instances that have not been solved by the respective presolvers are then used for training separate regression models per solver, which in turn are used for selecting a promising subset of “main solvers”.
The main difference between SATzilla2003 and SATzilla2007 lies in the regression models used for the main algorithm selection phase: While the former used empirical hardness models (LeytonBrown et al., 2002) based on ridge regression (see, e.g., Bishop, 2006), the latter employed hierarchical hardness models (Xu et al., 2007), more precisely
sparse multinomial logistic regression
(SMLR, see, e.g., Krishnapuram et al., 2005). The latter version of SATzilla sequentially runs up to two manually selected presolvers; if these fail to solve the given instance within a userspecified running time budget, instance features are computed and used to predict (and sequentially run) the best solver(s) from the given set . The system either terminates once the given SAT instance has been solved, or when a userspecified cutoff time is reached.Xu et al. (2008) compared different versions of SATzilla2007 on 2 300 random (RAND), 1 490 crafted/handmade (HAND) and 1 021 application/industrial (INDU) instances, using the same setup as the 2007 SAT Competition. For this purpose, they used a total of 48 features and identified between one and two presolvers, one backup solver, as well as three to five main solvers from a given set (which differed between RAND, HAND and INDU). Notably, their presolvers solved between 32 and 62% of the instances – despite running for only seven CPU seconds at most. Ultimately, SATzilla2007 ranked first (inter alia) in the SAT & UNSAT (RAND and HAND) categories of the competition, as well as in UNSAT (HAND).
3s.
Kadioglu et al. (2011) proposed a hybrid approach, denoted semistatic solver schedules (3S), which combines algorithm selection with solver scheduling. Since it can be very expensive to determine schedules over all solvers from a given set, Kadioglu et al. (2011) devised a different approach, in which they partitioned the (normalised) instance feature space for a given training set by means of means (Hamerly and Elkan, 2003). Then, the best value of to be used by the nearest neighbour algorithm (Hastie et al., 2009) was identified for each cluster and directly used for training semistatic solver schedules. 3S was demonstrated to perform very well; e.g., it reduced the gap between the (back then) stateoftheart selector on RAND instances, SATzilla2009_Rand, and the VBS by 57%. Additionally, this generalpurpose selector performed very well at the 2011 SAT Competition, winning seven medals (including two gold) without training separate selectors for different competition tracks (as had been done for previous versions of SATzilla).
SATzilla2012.
Building on the success of earlier versions of SATzilla, Xu et al. (2012) developed SATzilla2012, which showed outstanding performance in multiple SAT competitions. SATzilla2012
uses costsensitive pairwise classification as the basis for perinstance algorithm selection; these penalise incorrect predictions according to the loss in performance caused by them. More precisely, one costsensitive decision tree
(Ting, 2002) is used for every pair of solvers in the given set to predict which of the two solvers will perform better on the given instance. Simple voting over these pairwise predictions is used to determine the solver to be run. Like earlier versions of SATzilla, SATzilla2012 uses pre and backupsolvers in addition to the main algorithm selection stage. Additionally, a decision forest (Hastie et al., 2009) based on the number of clauses and variables in the given SAT instance is used to predict whether the computation of further instance features is sufficiently cheap to proceed to the main selection stage (otherwise, a statically chosen backup solver is run). In the 2012 SAT Competition, SATzilla2012 used a total of 31 SAT solvers and 138 features and ended up winning multiple prizes.Cshc.
Building on the static scheduling approach underlying 3S, Malitsky et al. (2013) introduced an algorithm selection system based on the core concept of
costsensitive hierarchical clustering
(CSHC). During its training phase, CSHCiteratively partitions the instance feature space by means of hyperplanes, and occasionally undoes splits if that leads to improvements in overall performance. When given a new instance,
CSHC first runs the 3S static algorithm schedule for 10% of the overall running time allotted for solving a given SAT instance and – in case remains unsolved – runs the SAT solver that performed best on the partition to which belongs. The resulting CSCH algorithm selection system has been reported to achieve even better performances than SATzilla2012 on a slightly modified version of the 2011 SAT competition (for details, see Malitsky et al., 2013).Snnap.
An approach dubbed solverbased nearest neigbour for algorithm portfolios (SNNAP; Collautti et al., 2013) successfully combines clustering with perinstance algorithm selection. It uses random forests (Hastie et al., 2009) to predict the running times of individual solvers. However, instead of directly selecting a solver based on these predictions, SNNAP uses them to identify SAT instances from the given training set that are similar to the instance to be solved. Specifically, instance similarity is quantified by means of the Jaccard distance – whose distance between two sets and is defined as – applied to binary vectors indicating a (small) fixed number of best solvers per instance. SNNAP then selects the solver that performed best on the nearest neighbours of the given instance, where is a userdefined constant. According to results reported by Collautti et al. (2013), despite the simplicity of the approach, SNNAP closes around 50% of the VBSSBS gap on a broad set of wellknown SAT instances.
AutoFolio.
In light of the many design choices encountered in the development of stateoftheart algorithm selection systems, Lindauer et al. (2015b, 2017a) proposed a powerful combination of perinstance algorithm selection and automated algorithm configuration: AutoFolio. In a nutshell, AutoFolio applies the automated algorithm configurator SMAC (Hutter et al., 2011) to the highly parametric algorithm selection framework claspfolio2 (Hoos et al., 2014). The space is structured in layers, starting with parameters for presolving schedules (including their allocated budgets), preprocessing procedures (transformations, filtering, etc.) and the algorithm selection systems (resembling a broad range of approaches, including SATzilla2007, SATzilla2012, 3S and SNNAP). Subsequent layers specify additional design decisions and (hyper)parameters. As demonstrated for multiple algorithm selection scenarios from the ASLib, AutoFolio indeed achieves results that are highly competitive with those of the bestperforming selection systems for a broad range of algorithm selection scenarios, without the need for manual choice of the selection mechanism or selector parameters (Lindauer et al., 2015b, 2017a).
AI planning.
The notion of algorithm selection can be applied to domains as well as to instances of AI planning problems. In the first case, a planner is selected for a specific domain and then applied for solving arbitrary instances in that domain. This is conceptually closely related to perset algorithm selection, as discussed in Section 2. In the second case, a planner is selected for a specific instance of a planning problem, such that even within the same domain, different planners may be chosen, depending on the characteristics of the specific problem instance. Unlike perdomain selection of planners, this is an instance of perinstance algorithm selection and hence will be our focus in the following.
Perdomain selection approaches.
Because perdomain selection of AI planners has been prominently studied in the literature, we briefly discuss some wellknown approaches. The PbP planning system and its successor, PbP2, are based on the idea of statistically analysing the performance of several domainindependent planning algorithms on a set of training instances from a given planning domain in order to select a set of planners and associated running times (Gerevini et al., 2009, 2011). When solving new instances from the same domain, these planners are run one after the other, using roundrobin scheduling with the predetermined running times for each planner. PbP and PbP2 also make use of macroactions, sequences of actions whose judicious use can considerably improve planner performance. PbP was the overall winner of the learning track of the 6th International Planning Competition, and PbP2 brought further improvement through the integration of automated algorithm configuration to better exploit the performance potential of parameterised planners.
The ASAP planning system is based on similar ideas (Vallati et al., 2013, 2014). In addition to macroactions, ASAP also exploits socalled entanglements, which reflect causal relationships that are characteristic for a given domain of planning problems. Different from PbP and PbP2, ASAP selects only a single representation (set of macroactions and entanglements) and planner for a given domain. On standard benchmarks, ASAP has been demonstrated to outperform PbP2 (as well as all the component planners it uses) in terms of the quality or cost of the plans found (Vallati et al., 2014).
IBaCoP2.
To the best of our knowledge, the first successful application of perinstance algorithm selection to AI planning was demonstrated by IBaCoP2 (Cenamor et al., 2014). IBaCoP2 uses 12 component planners, which were selected based on their performance on a large and diverse set of problem instances from past international planning competitions, applying Pareto efficiency analysis to the solution quality of the best plan found by a given planner, and the time required to find the first valid plan. A random forest model (see, e.g., Hastie et al., 2009), learned from performance data of the component planners on a set of training instances using WEKA (Witten et al., 2016)
, forms the core of the algorithm selection strategy. This model is used to predict whether a component planner will solve a given problem instance within a fixed time limit, based on a set of 35 cheaply computable, domainspecific features, some of which are derived from heuristics used in stateoftheart planning algorithms. To hedge against the consequences of poor predictions,
IBaCoP2 selects the five component planners with the highest estimated probability for solving the given instance and runs these, in the order of decreasing values of , one after the other, each for one fifth of the overall time given to the selector for solving . We note that, because of this latter strategy, IBaCoP2 is not a pure perinstance algorithm selector, but rather combines perinstance algorithm selection with a simple algorithm scheduling approach (cf. Section 2). IBaCoP and IBaCoP2 showed strong performance in the 8th International Planning Competition, with IBaCoP2 winning the sequential satisficing track (Vallati et al., 2015).Planzilla.
A second perinstance algorithm selection approach for AI planning has been considered by Rizzini et al. (2015, 2017). Their Planzilla system can be seen as an application of the previously outlined zilla approach (Cameron et al., 2017) to AI planning. Based on the default configuration of zilla, Planzilla is comprised of four sequential stages: (1) a static presolving schedule, (2) feature computation, (3) perinstance algorithm selection and (4) a backup solver. The presolving schedule is obtained by greedy selection from the given set of component planners and allocated 1/90 of the overall time budget for solving the given instance . Training instances solved during the presolving stage are not considered for constructing the perinstance selector, nor for selecting the backup solver. Perinstance algorithm selection makes use of a comprehensive set of 311 features that includes a broad range of properties of instance , as well as features derived from encoding into propositional satisfiability. Based on this set of features, Planzilla uses costsensitive classification forests for each pair of component planners in combination with a voting procedure to determine the planner to be run for the remainder of the given time budget. Before computing the complete feature set, which can be somewhat costly, Planzilla uses a simple model to predict whether feature computation can be completed within the remaining time, ; if not, feature computation and perinstance algorithm selection are skipped, and a backup solver is run instead. This backup solver is also run if the component planner selected in stage 3 terminates early without producing a valid plan; it is determined as the solver with the best performance for running time on the set of problem instances used to train Planzilla (excluding any instances solved during presolving).
Using all planners that participated in the optimal track of the 2014 International Planning Competition (IPC14), Planzilla was found to substantially outperform these individual planners and achieve performance close to that of the VBS (Rizzini et al., 2015, 2017). However, when evaluated on a set of testing instances dissimilar from those used for training, it was found that dynamic algorithm scheduling approaches performed better than Planzilla; these approaches dynamically construct an algorithm schedule by performing multiple stages of perinstance algorithm selection, using not only features of the planning instance to be solved, but also taking into account which component planners have already been run on , without success, in earlier stages of the schedule.
Travelling salesperson problem.
The potential for perinstance algorithm selection for the TSP differs markedly between exact TSP solvers, which are guaranteed to find provably optimal solutions for given TSP instances, and inexact solvers, which may find optimal solutions, but cannot produce a proof of optimality. In exact TSP solving, there is a single algorithm, Concorde (Applegate et al., 2007), that has defined stateoftheart performance for more than a decade. In contrast, for inexact TSP solving, there is no single algorithm that clearly dominates all others (across all types of instances). In fact, several studies (Pihera and Musliu, 2014; Kotthoff et al., 2015; Kerschke et al., 2017) have shown that at least three TSP solvers, EAX (Nagata and Kobayashi, 1997, 2013), LKH (Helsgaun, 2000, 2009) and MAOS (Xie and Liu, 2009), define stateoftheart performance on different kinds of TSP instances. In addition, two enhanced versions of EAX and LKH (denoted EAX+restart and LKH+restart), which employ additional restart mechanisms to overcome stagnation in the underlying search process, often (but not always) outperform EAX and LKH, respectively (DuboisLacoste et al., 2015). Further modifications of these algorithms – e.g., based on the alternative crossover operator proposed by Sanches et al. (2017a, b), which recently was integrated into LKH (Tinós et al., 2018) – might achieve even better performance; however, to this date, we are not aware of conclusive evidence to this effect. Instead, using automated algorithm selection techniques, the performance complementarity between existing solvers has been leveraged, leading to very substantial performance improvements over the single best solver (Kotthoff et al., 2015; Kerschke et al., 2017).
Kotthoff et al. (2015) compared EAX, LKH and their respective restart variants across four wellknown sets of TSP instances: random uniform Euclidean (RUE) instances, problems from the TSP library (TSPLIB), as well as national and VLSI instances^{2}^{2}2http://www.math.uwaterloo.ca/tsp/index.html. They used the feature sets proposed by Mersmann et al. (2013) and Hutter et al. (2014b) (see Section 3.1
) for constructing multiple algorithm selectors. Their best selector, based on multivariate adaptive regression splines
(MARS; see, e.g., Friedman, 1991), was trained on a predefined subset of features by Hutter et al. (2014b) and closed the gap between the single best solver from their set (EAX+restart) to the VBS by 10%.In an extended version of this earlier study, Kerschke et al. (2017)
considered additional types of TSP instances, feature sets and solvers, and furthermore employed more sophisticated machine learning techniques, including various feature selection strategies, for constructing algorithm selectors. Specifically, the set of TSP instances was extended by clustered and morphed instances
(Gent et al., 1999; Mersmann et al., 2012; Meisel et al., 2015), i.e., linear combinations of clustered and RUE instances^{3}^{3}3generated using the Rpackage netgen (Bossek, 2015). Furthermore, the basis for algorithm selection was expanded with the feature set of Pihera and Musliu (2014) and the MAOS solver by Xie and Liu (2009). The best algorithm selector under this extended setup was found to be a support vector machine
(Karatzoglou et al., 2004), which was constructed on a cheap, yet informative, subset of 16 nearestneighbour features (Pihera and Musliu, 2014). This particular selector achieved – despite its nonnegligible feature computation costs – a PAR10 score of 16.75s and thereby closed the gap between the single best solver (EAX+restart; 36.30s) and the virtual best solver (10.73s) by more than 75%.Further discrete combinatorial problems.
Throughout the previous paragraphs, we gave an overview of algorithm selection approaches for some of the most prominent and widely studied discrete combinatorial problems. Of course, perinstance algorithm selection has shown to be effective on several other discrete problems – such as the travelling thief problem (TTP), where Wagner et al. (2017) recently presented the first study of algorithm selection, along with a comprehensive collection of performance and feature data.
In some cases, successful applications of algorithm selection techniques have been described using different terminology. For instance, SmithMiles (2008) presented her algorithm selector for the quadratic assignment problem (QAP) under the umbrella of a “metalearning inspired framework”. Similarly, Pulina and Tacchella (2009) demonstrated the successful application of algorithm selection to the problem of solving quantified Boolean formulae (QBF), an important generalisation of SAT; yet, they describe their selector as a selfadaptive multiengine solver, which “selects among its reasoning engines the one which is more likely to yield optimal results”.
Considering the size of the literature on perinstance algorithm selection for discrete combinatorial problems, a comprehensive overview would be far beyond the scope of this survey and produce little additional insight. Instead, we will now shift our attention to continuous problems, which present different challenges and opportunities for algorithm selection.
5 Algorithm Selection for Continuous Problems
As previously mentioned, the efficacy of algorithm selection methods strongly depends on the performance data used for training them: The more representative the training set is regarding the entire range of possible problem instances, the better performance we can expect on previously unseen instances. For continuous optimisation problems, representativeness of benchmark sets has been a matter of longstanding debate, ranging from early works of De Jong (1975) up to more recent sets, e.g., from the CEC competitions (Li et al., 2013) or the blackbox optimisation benchmark (BBOB, Hansen et al., 2009) collection. There are several specific function generators, such as a framework for generating test functions with different degrees of multimodality (Wessing, 2015). Some of the most frequently used, and arguably most relevant test functions are included in the Pythonpackage optproblems (Wessing, 2016) and the Rpackage smoof (Bossek, 2017). While all these benchmark sets (and generators) have advantages and drawbacks, a detailed discussion is beyond the scope of this article. However, we note that the construction of representative training sets remains, at least to some degree, an open challenge in the context of algorithm selection for continuous optimisation problems.
Unconstrained singleobjective optimisation problems.
In singleobjective continuous optimisation, only few studies directly and successfully address the algorithm selection problem in an automated way. An initial approach of combining exploratory landscape analysis (ELA) and algorithm selection was presented by Bischl et al. (2012), focusing on the BBOB test suite. The latter consists of 24 functions which are grouped into four classes, mainly based on their multimodality, separability and global structure. Each function is represented by different instances resulting from slightly varied function parametrisations. Within the BBOB competition, 15 algorithm runs had to be conducted per function, equally distributed among five (BBOB 2009) or 15 (BBOB 2010) instances for decision space dimensions 2, 3, 5, 10, 20 and, optionally, 40. Algorithm performance was then evaluated using expected running time (ERT, Hansen et al., 2009), which reflects the expected number of function evaluations required to reach the global optimum up to a threshold of . Subsequently, the ERT was divided by the ERT of the best algorithm for this function within the respective competition to obtain a relative ERT indicator. Within the BBOB setup, accuracies in the range of are considered.
A representative set of four optimisation algorithms was constructed from the complete list of candidate solvers. Based on the lowlevel features introduced by Mersmann et al. (2011), Bischl et al. (2012)
aimed for an accurate prediction of the best of the four algorithms for each function within the benchmark set. For this purpose, a sophisticated costsensitive learning approach, based on onesided support vector regression with a radial basis function kernel, was used
(Tu and Lin, 2010). This complex approach enabled the minimisation of loss (measured by relative ERT) due to incorrect predictions. The median relative ERT of all 600 entries (five runs times five instances for each of the 24 BBOB functions) served as the overall performance measure of the resulting classifier. Two different crossvalidation strategies were investigated: crossvalidation over the instances of each given function or over the complete set of functions. The latter task can certainly be considered more challenging, due the structure of the BBOB test set, which was designed to comprise 24 functions with maximally diverse characteristics, covering a broad range of continous optimisation problems. While, as expected, better performance was observed in the first setting, performance was remarkably high in both cases.Recently, substantial progress has been made in terms of systematically constructing an automated algorithm selector for a joint dataset of all available BBOB test suites (Kerschke, 2017a; Kerschke and Trautmann, 2018), making use of the Rpackages flacco (Kerschke, 2017b, c), smoof (Bossek, 2017) and mlr (Bischl et al., 2016b). A total of 480 BBOB instances (instance IDs 1–5 of all 24 BBOB functions, across dimensions 2, 3, 5 and 10) were considered, combined with respective results of all 129 solvers submitted to the COCO platform (Hansen et al., 2016) so far. In order to keep the size of the set of solvers manageable and to focus on the most relevant and high performing solvers, the construction of the algorithm selector was based on a carefully selected subset of 12 solvers: two deterministic methods (variants of the BrentSTEP algorithm, BSrr and BSqi, see Baudiš and Pošík, 2015), five multilevel approaches – MLSL (Pál, 2013; Rinnooy Kan and Timmer, 1987), fmincon, fminunc, HMLSL (Pál, 2013) and MCS (Huyer and Neumaier, 2009) – as well as four CMAES variants: CMACSA (Atamna, 2015), IPOP400D (Auger et al., 2013), HCMA (Loshchilov et al., 2013) and SMACBBOB (Hutter et al., 2013). The commercial solver OptQuest/NLP (Pál, 2013; Ugray et al., 2007) was also included.
Performance was measured by relative ERT as in Bischl et al. (2012) by normalising the ERT for each solver per problem and dimension based on the best ERT for the respective problem (among the algorithms in the given set). A hybrid version of CMAES (HCMA, Loshchilov et al., 2013) turned out to be the single best solver (SBS), with a mean relative ERT score of 30.4 across all considered instances – and thus being the only solver to approximate the optimal objective value of all 96 problems up to the precision level of
used for this study. Various combinations of supervised learning (notably, classification, regression, pairwise regression) methods and feature selection strategies (greedy forwardbackward and backwardforward, two genetic algorithm variants) were utilised in combination with leaveonefunctionout crossvalidation. The best algorithm selector obtained in this manner, a classificationbased support vector machine
(Vapnik, 1995) combined with a sequence of a greedy forwardbackward and a geneticalgorithmbased feature selection approach, managed to reduce the mean relative ERT of the SBS roughly by half, to a value of 14.2, only requiring nine out of more than 300 exploratory landscape features. Specifically, metamodel and nearestbetter clustering features (see Section 3.2) were used in this context. Feature computation costs were taken into account and accounted for merely samples of the objective function (Kerschke et al., 2016a) – where denotes the dimensionality of the given decision space – which matches common intitial population sizes of evolutionary optimisation algorithms. Hence, when using such an evolutionary optimisation algorithm, making use of its initial population – which needs to be evaluated in any case – renders feature computation cost negligible.Constrained singleobjective optimisation problems.
In the area of constrained continuous optimisation, the performance of popular evolutionary computation techniques – such as differential evolution, evolution strategies and particle swarm optimisation – has been investigated for problems with linear and quadratic constraints (Poursoltan and Neumann, 2015b, a). Features capturing the correlation of these constraints have been investigated w.r.t. their impact on solver performance. Malan et al. (2015) and Malan and Moser (2018) numerically characterised constraint violations using landscape features. Based on results on the CEC 2010 benchmark problems, it was demonstrated that this approach produces detailed insights into constraint violation behaviour of continuous optimisation algorithms, indicating its potential usefulness in the context of algorithm selection and related approaches.
Furthermore, constraints have been evolved in different ways to construct instances that can be used for algorithm selection. This includes approaches maximising the performance difference of two given solvers, as well as a multiobjective approach for creating instances that reflect the tradeoffs between two given algorithms observed when varying the constraints. Neumann and Poursoltan (2016) demonstrated that the multiobjective approach leads to an instance set that provides a better basis for algorithm selection than sets obtained by maximising performance differences.
Multiobjective optimisation problems.
So far, there are no systematic studies of automated algorithm selection for multiobjective continuous optimisation problems, as feature design is already extremely challenging and the suitability of existing benchmark sets is questionable. However, initial approaches regarding multiobjective features, landscape analysis and multimodality exist and offer promising perspectives for future research (see Section 6).
6 Perspectives and Open Problems
In the previous sections, we have given an overview of the stateoftheart in algorithm selection for discrete and continuous problems. We have summarised the general approach of perinstance algorithm selection and discussed a number of related problems, including perset algorithm selection, automated algorithm configuration, algorithm schedules and parallel algorithm portfolios. An important aspect for any perinstance algorithm selection approach is the design of informative and cheaply computable features that are able to characterise and differentiate problem instances w.r.t. a given set of algorithms. We have given an overview of feature sets for several prominent discrete problems, as well as features used in continuous blackbox optimisation. Based on this, we have summarised prominent algorithm selection approaches for discrete and continuous problems. In the following, we will discuss perspectives and challenges for future research in the area of automated algorithm selection.
Performance measures.
A crucial part of any empirical performance evaluation – which provides the basis for constructing algorithm selectors – is the underlying performance measure. While penalised average running time (notably, PAR10, Bischl et al., 2016a) and expected running time (ERT, Hansen et al., 2009)
are commonly used in this context, some of its parameters can substantially affect performance measurements. For example, in the case of PAR10, the penalty factor is set to 10 and an arithmetic mean is used to aggregate over multiple runs or problem instances. Both choices can, in principle, be varied (e.g., by replacing the arithmetic mean by different quantiles), with significant effects on the robustness of solvers selected based on them.
Recently, Kerschke et al. (2018a) presented a structured approach on how to assess the sensitivity of an empirical performance evaluation w.r.t. altered requirements regarding solver robustness across runs, focusing on PAR10 and ERT. As demonstrated within this study, by adjusting the parameters of the performance measures, users are able to adapt the ranking of a given set of algorithms, and the performance characteristics of the algorithm selectors constructed based on that set, according to their preferences – trading off, for example, running time vs. robustness. In an alternative approach, van Rijn et al. (2017) utilise the advantages of two popular performance measures – ERT and fixed cost error (FCE, see, e.g. Bäck et al., 2013) – by combining and standardising them within a joint performance measure. Moreover, a multiobjective perspective on performance measurement shows promise, e.g., by enabling direct investigations of the tradeoff between the number of failed runs and the average running time of successful runs of a given solver (Bossek and Trautmann, 2018). Concepts such as Paretooptimality and related multiobjective quality indicators (Coello Coello et al., 2007) could then be used in the context of constructing and assessing the performance of algorithm selectors and related metaalgorithmic techniques.
Evolving / generating problem instances.
Naturally, an algorithm selector only generalises to problems which are similar enough to the instances contained in the benchmark set that was used to construct it. Usually, common benchmark sets are considered, which are deemed to be representative of a specific use context. However, one could argue that benchmark sets should be designed specifically w.r.t. the given set of candidate solvers, such that they exhibit maximum diversity regarding the challenges posed by the instances for the specific solvers. The idea of evolving instances utilising an evolutionary algorithm dates back to
van Hemert (2006) and SmithMiles et al. (2010), who constructed instances that are extremely hard or easy for specific TSP solvers; these works were followed by more sophisticated approaches of Mersmann et al. (2012, 2013) and Nallaperuma et al. (2013). Bossek and Trautmann (2016a, b) further explored this idea by evolving TSP instances that maximise the performance difference between two given solvers, i.e., instances that are extremely hard to solve for one solver, but very easy for the other. This can, in principle, provide insights into links between performance differences and instance characteristics – a topic that is highly relevant for automated algorithm selection.Recent studies build upon these concepts by explicitly focusing on the diversity of evolved instances (Gao et al., 2015; Neumann et al., 2018), paving the way for a systematic approach to construct most informative and relevant benchmarks specifically tailored to a given set of solvers. The next promising step in this direction will be the complementation of stateoftheart benchmarks with those specifically designed instances in order to provide most informative benchmark sets tailored to given sets of solvers. Such sets are likely to provide a basis for constructing perinstance algorithm selectors whose performance generalises better to problem instances that differ from those used during their construction. Improvements, configuration and enhancements of the underlying evolutionary algorithm offer extremely promising research perspectives. Interestingly, the overall approach also provides a way for systematically detecting advantages and shortcomings of specific solvers, and thus produce benefits for the analysis and design of algorithms beyond algorithm selection and related approaches.
Online algorithm selection.
The work covered in this survey is mainly related to (static) offline algorithm selection, where algorithm selectors are constructed based on using a set of training instances prior to applying them to new problem instances. Yet, according to Armstrong et al. (2006), Gagliolo and Schmidhuber (2010) and Degroote et al. (2016), in principle, it might be possible to obtain even better results using (dynamic) online algorithm selection methods, which adapt an algorithm selector while it is being used to solve a series of problem instances. Although this involves a certain overhead, it can enable better performance and increased robustness, as the selector can react better to changes in the stream of problem instances it is tasked to solve. In order to more easily amortise the overhead involved in online algorithm selection, building on earlier work on probing features (Hutter et al., 2014b; Kotthoff et al., 2015), the development of cheap and informative monitoring features – i.e., features that extract sufficient instancespecific information without significantly reducing solver or selector performance – is likely to be of key importance.
Another approach for online algorithm selection and the selection of an appropriate algorithm for a given problem instance is provided by socalled hyperheuristics; these are algorithms for selecting or generating solvers for a given problem from a set of given heuristic components (Burke et al., 2013). In many cases, they employ heuristic, rulebased mechanisms and make use of rather simple components, such as greedy construction algorithms for the given problem.
Lifelong learning hyperheuristics are applied in a setting where a series of problem instances is solved. Based on the performance of previously chosen component solvers, a solver for a new problem instance is chosen that is deemed most likely to perform best. Lifelong learning hyperheuristics have achieved good results for wellknown decision problems, such as constraint satisfaction (OrtizBayliss et al., 2015) and bin packing (Sim et al., 2015).
Features for mixed (discrete + continuous) problems.
Developing good features for a given problem is a challenging task that provides a crucial basis for effective algorithm selection techniques. As discussed previously, rich sets of features have been introduced for wellstudied discrete and continuous problems, but many combinatorial problems of practical importance involve discrete and continuous decision variables. Perhaps the best example for this is mixed integer programming (MIP), a problem of great importance in the context of a broad range of challenging realworld optimisation tasks. Hutter et al. (2014b) introduced 95 features for MIP, including problem type and size, variableconstraint graph, linear constraint matrix, objective function and LPbased features. For the travelling thief problem (TTP), which can be seen as a combination of the TSP and knapsack problem (KP), algorithm selection has been studied by Wagner et al. (2017). They used 48 TSP and four KP features, plus three parameters of the TTP that connect the TSP and KP parts of the problem. Features for the KP that characterise the correlation of weights and profits have so far not been taken into account, although they seem to provide a further improvement in the characterisation of TTP instances.
Algorithm selection for multiobjective optimisation problems.
While we have covered numerous studies on algorithm selection in this survey, none of them has dealt with multiobjective optimisation problems. From a practitioner’s point of view, this is a significant limitation, as multiple competing objectives arise in many, perhaps most, realworld problems^{4}^{4}4Consider, for example, the tradeoff between the performance improvement and the accompanying costs for finding such an improved solution.. However, for convenience, these problems are often handled as singleobjective problems – e.g., by focusing on the most important objective or by applying scalarisation functions. Many prominent benchmarks for continuous multiobjective optimisation algorithms, such as biobjective BBOB (Tušar et al., 2016), DTLZ (Deb et al., 2005), ED (Emmerich and Deutz, 2007), MOP (van Veldhuizen, 1999), UF (Zhang et al., 2008), WFG (Huband et al., 2006) and ZDT (Zitzler et al., 2000), are entirely artificial, and it is unclear to which degree they resemble realworld problems.
In addition, there is a dearth of research on the characterisation of those optimisation problems – in particular by means of automatically computable features. Of course, one could compute variants of existing features for each of the objectives separately (see, e.g., Kerschke and Trautmann, 2016), but this completely ignores interaction effects between the objectives, which in turn have a strong impact on the landscapes (even for rather simple problems, see, e.g., Kerschke and Grimme, 2017). Hence, there is a significant need and opportunity for research on visualising multiobjective landscapes (see, e.g., da Fonseca, 1995; Tušar, 2014; Tušar and Filipič, 2015; Kerschke and Grimme, 2017), as well as characterising them (numerically) – along the lines of Ulrich et al. (2010), Kerschke et al. (2016b, 2018b) or Grimme et al. (2018) – as this will (a) improve the understanding of multiobjective problems and their specific properties, and (b) provide a basis for automated feature computation. We expect the latter to be of key importance for the development of new algorithms and effective perinstance selectors in the area of multiobjective optimisation.
In order to construct a powerful and complementary portfolio of multiobjective optimisers (a priori), as well as for analysing the strengths and weaknesses of the resulting algorithm selector (a posteriori), visual approaches such as the empirical attainment function (EAF) plots (da Fonseca et al., 2005; da Fonseca and da Fonseca, 2010; LópezIbáñez et al., 2010) provide valuable feedback on location, spread and interpoint dependence structure of the considered optimisers’ Pareto set approximations.
Algorithm selection on streaming data.
The importance of automated algorithm selection and related approaches in the context of learning on streaming data should not be neglected. Streaming data (Bifet et al., 2018) pose considerable challenges for the respective algorithms, as (a) data points arrive as a constant stream, (b) the size of the stream is large and potentially unbounded, (c) the order of data points cannot be influenced, (d) data points can typically only be evaluated once and are discarded afterwards, and (e) the underlying distribution of the data points in the stream can change over time (nonstationarity or concept drift).
Few concepts for automated algorithm selection on streaming data exist so far, both for supervised (see, e.g., van Rijn et al., 2014, 2018)
and unsupervised learning algorithms. In unsupervised learning, stream clustering is a very active research field. Although several stream clustering approaches exist
(see surveys of Amini et al., 2014; Carnein and Trautmann, 2018; Mansalis et al., 2018), these have many parameters that affect their performance, yet clear guidelines on how to set and adjust them over time are lacking. Moreover, different kinds of algorithms are required in the socalled online phase (maintaining an informative aggregated data stream representation, in terms of microclusters) and offline phase (standard clustering algorithm, such as means applied to the microclusters), which leads to a huge space of parameter and algorithm combinations. An initial approach on configuring and benchmarking stream clustering approaches based on irace (LópezIbáñez et al., 2016) has been presented by Carnein et al. (2017). Building on this work, especially in light of remark (e) above, we expect that algorithm selection will play an important role for robust learning on streaming data – especially when keeping the efficiency regarding realtime capability in mind. This will require informative feature sets, ensemble techniques, as well as a much wider range of suitable benchmarks. Of course, ideally a combination of algorithm selection and (online) configuration is desired and a very promising line of research.Perinstance algorithm configuration.
As previously explained, automatic algorithm configuration involves the determination of parameter settings of a given target algorithm to optimise its performance on a given set of problem instances. Perinstance algorithm configuration (PIAC) is a variant of this problem in which parameter settings are determined for a given problem instance to be solved. It can be seen as a generalisation of perinstance algorithm selection, in which the set of algorithms that form the basis for selection comprises all (valid) configurations of a single, parameterised algorithm. Analogous to perinstance algorithm selection, PIAC involves two phases: an offline phase, during which a perinstance configurator is trained, and an online phase, in which this configurator is run on given problem instances. During the latter, a configuration is determined based on features of the problem instance to be solved. Standard, perset algorithm configuration, in contrast, is an offline process that results in a single configuration that is subsequently used to solve problem instances presumed to resemble those used during training. PIAC is challenging, because the spaces of (valid) configurations to select from is typically very large (see, e.g., Hutter et al., 2014a), and compared to the size of these configuration spaces, any training data used during the offline construction of the perinstance configurator is necessarily sparse. In particular, for typical configuration scenarios, the training data would necessarily cover only a very small number of configurations, which makes it challenging to learn a mapping from instance features to configurations.
We consider PIAC to be a largely open problem, with significant potential for future work. There is some evidence in the literature that it may have significant benefits compared to the more established perset configuration techniques. Notably, Kadioglu et al. (2010) proposed a PIAC approach based on a combination of clustering and a standard, perset algorithm configurator and reported promising results on several set covering, mixed integer programming and propositional satisfiability algorithms.
Further challenges.
Due to the steady stream of work in algorithm selection and related areas, it is important to keep track of promising developments. Of course, domainrelated research networks such as COSEAL^{5}^{5}5https://www.coseal.net/ might relieve this challenging task to some degree, yet they will not be able to keep up with all developments within this fastgrowing and productive community. Instead, comparisons against stateoftheart methods, which are of special significance in this context, are facilitated by benchmarking platforms and libraries, such as ASLib (Bischl et al., 2016a), ACLib (Hutter et al., 2014a) and HPOlib (Eggensperger et al., 2013) for algorithm selection, configuration and hyperparameter optimisation, respectively. At the same time, it is important (a) to promote and establish as best practice the use of these libraries, especially in the context of newly proposed methods for algorithm selection and related problems, and (b) to maintain and expand these libraries, in order to ensure their continued relevance, e.g., by integrating scenarios for multiobjective and additional realworld problems.
The latter not only applies to the previously mentioned libraries, but also to broader benchmark collections for the underlying specific problems. For example, recent studies have analysed the ‘footprints’ of different continuous optimisation algorithms on common benchmarks; while Muñoz Acosta and SmithMiles (2017) focused on BBOB (arguably the most prominent benchmark in continuous blackbox optimisation), Muñoz Acosta et al. (2018) applied a similar analysis to machine learning problems from the UCI repository (Dheeru and Karra Taniskidou, 2017) and OpenML (van Rijn et al., 2013). Their visual analyses indicate that (a) different “comfort zones” for the various algorithms in question exist across the respective instance spaces, in line with what might be expected based on a liberal interpretation of the NFL theorems by Wolpert and Macready (1997), and (b) the instances from common benchmarks’ problems in continuous optimisation are not very diverse, but cover only relatively small areas of the overall problem instance space.
Another important direction for future work is the improvement of problemspecific features in general. Aside from the directions outlined previously (monitoring features as well as features for mixed and multiobjective problems), more informative and cheaper features are always desirable and likely to pave the way towards more effective applications of algorithm selection and related techniques.
An interesting open question regards the tradeoff between the performance achieved by algorithm selection approaches, e.g., in relation to a hypothetical perfect selector (VBS), and their complexity, including the complexity of the feature sets they operate on. There is recent evidence from an application of algorithm selection to solvers for quantified Boolean formulae (QBF) that suggests that sometimes, a small number of simple features is sufficient for achieving excellent performance (Hoos et al., 2018). However, it is presently unclear to which extent this situation arises in other application scenarios, and to which degree it is contingent on the use of highly sophisticated algorithm selection techniques.
Finally, an intriguing direction for future work is the development of algorithm selection techniques for automated algorithm configurators and selectors. Intuitively, it is clear that different algorithm configuration scenarios would be handled most efficiently using rather different configuration procedures (depending, e.g., on the prevalence of numerical vs. categorical parameters). Likewise, it has been observed in the recent Open Algorithm Selection Competition (Lindauer et al., 2018) that different AS techniques work best on different AS scenarios – suggesting that metaalgorithm selection (i.e., AS applied to AS strategies) might be useful for quickly identifying the selection strategy to be used in a particular application context. In both cases, configurator selection and metaselection, the limited amount of training data is likely to give rise to specific challenges, which may well require the development of new AS techniques.
Acknowledgements
Pascal Kerschke, Heike Trautmann and Holger H. Hoos acknowledge support from the European Research Center for Information Systems (ERCIS). The former two also acknowledge support from the DAAD PPP project No. 57314626. Frank Neumann acknowledges the support of the Australian Research Council through grant DP160102401. The authors gratefully acknowledge useful inputs from Lars Kotthoff and Jakob Bossek.
References
 Abell et al. (2013) Abell, T., Malitsky, Y., and Tierney, K. (2013). Features for Exploiting BlackBox Optimization Problem Structure. In Nicosia, G. and Pardalos, P., editors, Proceedings of the 7th International Conference on Learning and Intelligent Optimization (LION), volume 7997 of Lecture Notes in Computer Science (LNCS), pages 30 – 36. Springer.
 Amini et al. (2014) Amini, A., Wah, T. Y., and Saboohi, H. (2014). On DensityBased Data Streams Clustering Algorithms: A Survey. Journal of Computer Science and Technology (JCST), 29(1):116 – 141.
 Angel and Zissimopoulos (2002) Angel, E. and Zissimopoulos, V. (2002). On the Hardness of the Quadratic Assignment Problem with Metaheuristics. Journal of Heuristics, 8(4):399 – 414.
 Ansótegui et al. (2015) Ansótegui, C., Malitsky, Y., Samulowitz, H., Sellmann, M., and Tierney, K. (2015). ModelBased Genetic Algorithms for Algorithm Configuration. Proceedings of the TwentyFourth International Joint Conference on Artificial Intelligence (IJCAI), pages 733 – 739.
 Applegate et al. (2007) Applegate, D. L., Bixby, R. E., Chvátal, V., and Cook, W. J. (2007). The Traveling Salesman Problem: A Computational Study. Princeton University Press, Princeton, NJ, USA.
 Arik et al. (2015) Arik, S., Huang, T., Lai, W. K., and Liu, Q., editors (2015). Proceedings Part III of the 22nd International Conference on Neural Information Processing (ICONIP), volume 9491 of Lecture Notes in Computer Science (LNCS). Springer.

Armstrong et al. (2006)
Armstrong, W., Christen, P., McCreath, E., and Rendell, A. P. (2006).
Dynamic Algorithm Selection Using Reinforcement Learning.
In International Workshop on Integrating AI and Data Mining (AIDM), pages 18 – 25. IEEE.  Atamna (2015) Atamna, A. (2015). Benchmarking IPOPCMAESTPA and IPOPCMAESMSR on the BBOB Noiseless Testbed. In Proceedings of the 17th Annual Conference on Genetic and Evolutionary Computation (GECCO), pages 1135 – 1142, New York, NY, USA. ACM.
 Auger et al. (2013) Auger, A., Brockhoff, D., and Hansen, N. (2013). Benchmarking the Local Metamodel CMAES on the Noiseless BBOB’2013 Test Bed. In Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation (GECCO), pages 1225 – 1232. ACM.
 Bäck et al. (2013) Bäck, T., Foussette, C., and Krause, P. (2013). Contemporary Evolution Strategies. Natural Computing Series. Springer.
 Bagheri et al. (2017) Bagheri, S., Konen, W., Allmendinger, R., Branke, J., Deb, K., Fieldsend, J., Quagliarella, D., and Sindhya, K. (2017). Constraint Handling in Efficient Global Optimization. In Proceedings of the 19th Annual Conference on Genetic and Evolutionary Computation (GECCO), pages 673 – 680. ACM.
 Baral (2003) Baral, C. (2003). Knowledge Representation, Reasoning and Declarative Problem Solving. Cambridge University Press.

Baudiš and Pošík (2015)
Baudiš, P. and Pošík, P. (2015).
Global Line Search Algorithm Hybridized with Quadratic Interpolation and its Extension to Separable Functions.
In Proceedings of the 17th Annual Conference on Genetic and Evolutionary Computation (GECCO), pages 257 – 264, New York, NY, USA. ACM.  Belkhir et al. (2016) Belkhir, N., Dréo, J., Savéant, P., and Schoenauer, M. (2016). Feature Based Algorithm Configuration: A Case Study with Differential Evolution. In Handl, J., Hart, E., Lewis, P. R., LópezIbáñez, M., Ochoa, G., and Paechter, B., editors, Proceedings of the 14th International Conference on Parallel Problem Solving from Nature (PPSN XIV), volume 9921 of Lecture Notes in Computer Science (LNCS), pages 156 – 166. Springer.
 Belkhir et al. (2017) Belkhir, N., Dréo, J., Savéant, P., and Schoenauer, M. (2017). Per Instance Algorithm Configuration of CMAES with Limited Budget. In Proceedings of the 19th Annual Conference on Genetic and Evolutionary Computation (GECCO), pages 681 – 688. ACM.
 Biere et al. (2009) Biere, A., Heule, M., van Maaren, H., and Walsh, T. (2009). Handbook of Satisfiability, volume 185. IOS Press.
 Bifet et al. (2018) Bifet, A., Gavaldà, R., Holmes, G., and Pfahringer, B. (2018). Machine Learning for Data Streams with Practical Examples in MOA. MIT Press.
 Birattari et al. (2002) Birattari, M., Stützle, T., Paquete, L., and Varrentrapp, K. (2002). A Racing Algorithm for Configuring Metaheuristics. In Proceedings of the 4th Annual Conference on Genetic and Evolutionary Computation (GECCO), pages 11 – 18.
 Bischl et al. (2016a) Bischl, B., Kerschke, P., Kotthoff, L., Lindauer, T. M., Malitsky, Y., Fréchette, A., Hoos, H. H., Hutter, F., LeytonBrown, K., Tierney, K., and Vanschoren, J. (2016a). ASlib: A Benchmark Library for Algorithm Selection. Artificial Intelligence (AIJ), 237:41 – 58.
 Bischl et al. (2016b) Bischl, B., Lang, M., Kotthoff, L., Schiffner, J., Richter, J., Studerus, E., Casalicchio, G., and Jones, Z. M. (2016b). mlr: Machine Learning in R. Journal of Machine Learning Research (JMLR), 17(170):1 – 5.
 Bischl et al. (2012) Bischl, B., Mersmann, O., Trautmann, H., and Preuss, M. (2012). Algorithm Selection Based on Exploratory Landscape Analysis and CostSensitive Learning. In Proceedings of the 14th Annual Conference on Genetic and Evolutionary Computation (GECCO), pages 313 – 320. ACM.
 Bishop (2006) Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer.
 Bossek (2015) Bossek, J. (2015). netgen: Network Generator for Combinatorial Graph Problems. Rpackage version 1.0.
 Bossek (2017) Bossek, J. (2017). smoof: Single and MultiObjective Optimization Test Functions. The R Journal.
 Bossek et al. (2018) Bossek, J., Grimme, C., Meisel, S., Rudolph, G., and Trautmann, H. (2018). Local Search Effects in BiObjective Orienteering. In Proceedings of the 20th Annual Conference on Genetic and Evolutionary Computation (GECCO), pages 585 – 592. ACM.
 Bossek and Trautmann (2016a) Bossek, J. and Trautmann, H. (2016a). Evolving Instances for Maximizing Performance Differences of StateofTheArt Inexact TSP Solvers. In Festa, P., Sellmann, M., and Vanschoren, J., editors, Proceedings of the 10th International Conference on Learning and Intelligent Optimization (LION), volume 10079 of Lecture Notes in Computer Science (LNCS), pages 48 – 59, Ischia, Italy. Springer.
 Bossek and Trautmann (2016b) Bossek, J. and Trautmann, H. (2016b). Understanding Characteristics of Evolved Instances for StateoftheArt Inexact TSP Solvers with Maximum Performance Difference. In Adorni, G., Cagnoni, S., Gori, M., and Maratea, M., editors, Proceedings of the Conference of the Italian Association for Artificial Intelligence (AI*IA 2016), volume 10037 of Lecture Notes in Computer Science, pages 3 – 12.
 Bossek and Trautmann (2018) Bossek, J. and Trautmann, H. (2018). MultiObjective Performance Measurement: Alternatives to PAR10 and Expected Running Time. In Proceedings of the 4th International Conference on Learning and Intelligent Optimization (LION), Lecture Notes in Computer Science (LNCS). Springer. Publication status: Accepted.
 Boukeas et al. (2004) Boukeas, G., Halatsis, C., Zissimopoulos, V., and Stamatopoulos, P. (2004). Measures of Intrinsic Hardness for Constraint Satisfaction Problem Instances. In Van Emde Boas, P., Pokorný, J., Bieliková, M., and Štuller, J., editors, SOFSEM: 30th International Conference on Current Trends in Theory and Practice of Computer Science, volume 2932 of Lecture Notes in Computer Science (LNCS), pages 184 – 195. Springer.
 Brooks and Durfee (2003) Brooks, C. H. and Durfee, E. H. (2003). Using Landscape Theory to Measure Learning Difficulty for Adaptive Agents. In Adaptive Agents and MultiAgent Systems, volume 2636 of Lecture Notes in Computer Science (LNCS), pages 291 – 305. Springer.
 Burke et al. (2013) Burke, E. K., Gendreau, M., Hyde, M. R., Kendall, G., Ochoa, G., Özcan, E., and Qu, R. (2013). HyperHeuristics: A Survey of the State of the Art. Journal of the Operational Research Society (JORS), 64(12):1695 – 1724.
 Cameron et al. (2017) Cameron, C., Hoos, H. H., LeytonBrown, K., and Hutter, F. (2017). OASC2017: Zilla Submission. In Lindauer, T. M., van Rijn, J. N., and Kotthoff, L., editors, Proceedings of the Open Algorithm Selection Challenge, volume 79 of Proceedings of Machine Learning Research, pages 15–18, Brussels, Belgium.
 Carnein et al. (2017) Carnein, M., Assenmacher, D., and Trautmann, H. (2017). An Empirical Comparison of Stream Clustering Algorithms. In Proceedings of the Computing Frontiers Conference, pages 361 – 365. ACM.
 Carnein and Trautmann (2018) Carnein, M. and Trautmann, H. (2018). Optimizing Data Stream Representation: An Extensive Survey on Stream Clustering Algorithms. Business and Information Systems Engineering (BISE). Publication status: Accepted.
 Cenamor et al. (2013) Cenamor, I., de la Rosa, T., and Fernández, F. (2013). Learning Predictive Models to Configure Planning Portfolios. In Proceedings of the Fourth Workshop on Planning and Learning at the TwentyThird International Conference on Automated Planning and Scheduling (ICAPSPAL), pages 14 – 22.
 Cenamor et al. (2014) Cenamor, I., de la Rosa, T., and Fernández, F. (2014). IBaCoP and IBaCoP2 Planner. Proceedings of the Eighth International Planning Competition (IPC), pages 35 – 38.
 Coello Coello et al. (2007) Coello Coello, C. A., Lamont, G. B., and van Veldhuizen, D. A. (2007). Evolutionary Algorithms for Solving MultiObjective Problems. Springer.
 Collautti et al. (2013) Collautti, M., Malitsky, Y., Mehta, D., and O’Sullivan, B. (2013). SNNAP: SolverBased Nearest Neighbor for Algorithm Portfolios. In Blockeel, H., Kersting, K., Nijssen, S., and Železnỳ, F., editors, Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD), volume 8190 of Lecture Notes in Computer Science (LNCS), pages 435 – 450. Springer.
 Cook and Varnell (1997) Cook, D. J. and Varnell, R. C. (1997). Maximizing the Benefits of Parallel Search Using Machine Learning. In Proceedings of the Fourteenth National Conference on Artificial Intelligence (AAAI 1997), pages 559 – 564. Association for the Advancement of Artificial Intelligence (AAAI).
 Culberson (1998) Culberson, J. C. (1998). On the Futility of Blind Search: An Algorithmic View of “No Free Lunch”. Evolutionary Computation (ECJ), 6(2):109 – 127.
 da Fonseca (1995) da Fonseca, C. M. M. (1995). Multiobjective Genetic Algorithms with Application to Control Engineering Problems. PhD Thesis, Department of Automatic Control and Systems Engineering, University of Sheffield.
 da Fonseca et al. (2005) da Fonseca, C. M. M., da Fonseca, V. G., and Paquete, L. (2005). Exploring the Performance of Stochastic Multiobjective Optimisers with the SecondOrder Attainment Function. In Coello Coello, C. A., Aguirre, A. H., and Zitzler, E., editors, Proceedings of the 3rd International Conference on Evolutionary MultiCriterion Optimization (EMO), volume 3410 of Lecture Notes in Computer Science (LNCS), pages 250 – 264. Springer.
 da Fonseca and da Fonseca (2010) da Fonseca, V. and da Fonseca, C. M. M. (2010). The AttainmentFunction Approach to Stochastic Multiobjective Optimizer Assessment and Comparison. In BartzBeielstein, T., Chiarandini, M., Paquete, L., and Preuss, M., editors, Experimental Methods for the Analysis of Optimization Algorithms, pages 103 – 130. Springer.
 Davidor (1991) Davidor, Y. (1991). Epistasis Variance: A Viewpoint on GAHardness. In Foundations of Genetic Algorithms, volume 1, pages 23 – 35. Elsevier.
 De Jong (1975) De Jong, K. A. (1975). Analysis of the Behavior of a Class of Genetic Adaptive Systems. PhD Thesis, University of Michigan.
 Deb et al. (2005) Deb, K., Thiele, L., Laumanns, M., and Zitzler, E. (2005). Scalable Test Problems for Evolutionary Multiobjective Optimization. In Abraham, A., Jain, L., and Goldberg, R., editors, Evolutionary Multiobjective Optimization, Advanced Information and Knowledge Processing (AI & KP), pages 105 – 145. Springer.
 Degroote et al. (2016) Degroote, H., Bischl, B., Kotthoff, L., and De Causmaecker, P. (2016). Reinforcement Learning for Automatic Online Algorithm Selection – an Empirical Study. In Brejová, B., editor, Proceedings of ITAT 2016: Information Technologies – Applications and Theory: Conference on Theory and Practice of Information Technologies, volume 1649, pages 93 – 101.
 Dheeru and Karra Taniskidou (2017) Dheeru, D. and Karra Taniskidou, E. (2017). UCI Machine Learning Repository.
 Dietterich (2000) Dietterich, T. G. (2000). Ensemble Methods in Machine Learning. In Multiple Classifier Systems, volume 1857 of Lecture Notes in Computer Science (LNCS), pages 1 – 15. Springer.
 DuboisLacoste et al. (2015) DuboisLacoste, J., Hoos, H. H., and Stützle, T. (2015). On the Empirical Scaling Behaviour of Stateoftheart Local Search Algorithms for the Euclidean TSP. In Proceedings of the 17th Annual Conference on Genetic and Evolutionary Computation (GECCO), pages 377 – 384, New York, NY, USA. ACM.
 Eggensperger et al. (2013) Eggensperger, K., Feurer, M., Hutter, F., Bergstra, J., Snoek, J., Hoos, H. H., and LeytonBrown, K. (2013). Towards an Empirical Foundation for Assessing Bayesian Optimization of Hyperparameters. In NIPS Workshop on Bayesian Optimization in Theory and Practice, volume 10.
 Emmerich and Deutz (2007) Emmerich, M. T. M. and Deutz, A. H. (2007). Test Problems Based on Lamé Superspheres. In Obayashi, S., Deb, K., Poloni, C., Hiroyasu, T., and Murata, T., editors, Proceedings of the 4th International Conference on Evolutionary MultiCriterion Optimization (EMO), volume 4403 of Lecture Notes in Computer Science (LNCS), pages 922 – 936. Springer.
 Fawcett et al. (2014) Fawcett, C., Vallati, M., Hutter, F., Hoffmann, J., Hoos, H. H., and LeytonBrown, K. (2014). Improved Features for Runtime Prediction of DomainIndependent Planners. In Proceedings of the TwentyFourth International Conference on Automated Planning and Scheduling (ICAPS). Association for the Advancement of Artificial Intelligence (AAAI).
 Flamm et al. (2002) Flamm, C., Hofacker, I. L., Stadler, P. F., and Wolfinger, M. T. (2002). Barrier Trees of Degenerate Landscapes. Zeitschrift für Physikalische Chemie. International Journal of Research in Physical Chemistry and Chemical Physics, 216(2/2002):155 – 173.
 Fonlupt et al. (1998) Fonlupt, C., Robilliard, D., and Preux, P. (1998). A BitWise Epistasis Measure for Binary Search Spaces. In Eiben, Á. E., Bäck, T. H. W., Schoenauer, M., and Schwefel, H.P., editors, Proceedings of the 5th International Conference on Parallel Problem Solving from Nature (PPSN V), volume 1498 of Lecture Notes in Computer Science (LNCS), pages 47 – 56. Springer.
 Friedman (1991) Friedman, J. H. (1991). Multivariate Adaptive Regression Splines. The Annals of Statistics, pages 1 – 67.
 Fukunaga (2000) Fukunaga, A. S. (2000). Genetic Algorithm Portfolios. In Proceedings of the IEEE Congress on Evolutionary Computation (CEC), pages 1304 – 1311. IEEE.
 Gagliolo and Schmidhuber (2010) Gagliolo, M. and Schmidhuber, J. (2010). Algorithm selection as a bandit problem with unbounded losses. In Blum, C. and Battiti, R., editors, Proceedings of the 4th International Conference on Learning and Intelligent Optimization (LION), volume 6073 of Lecture Notes in Computer Science (LNCS), pages 82 – 96. Springer.
 Gao et al. (2015) Gao, W., Nallaperuma, S., and Neumann, F. (2015). FeatureBased Diversity Optimization for Problem Instance Classification. arXiv preprint, abs/1510.08568. Conference version appeared in PPSN 2016.
 Gent et al. (1999) Gent, I. P., Hoos, H. H., Prosser, P., and Walsh, T. (1999). Morphing: Combining Structure and Randomness. In Proceedings of the Sixteenth National Conference on Artificial Intelligence (AAAI 1999), volume 99, pages 654 – 660. Association for the Advancement of Artificial Intelligence (AAAI).
 Gerevini and Long (2005) Gerevini, A. and Long, D. (2005). Plan Constraints and Preferences in PDDL3. Technical report, Department of Electronics for Automation, University of Brescia.
 Gerevini et al. (2003) Gerevini, A., Saetti, A., and Serina, I. (2003). Planning Through Stochastic Local Search and Temporal Action Graphs in LPG. Journal of Artificial Intelligence Research (JAIR), 20:239 – 290.
 Gerevini et al. (2009) Gerevini, A., Saetti, A., and Vallati, M. (2009). An Automatically Configurable Portfoliobased Planner with MacroActions: PbP. In Proceedings of the Nineteenth International Conference on Automated Planning and Scheduling (ICAPS). Association for the Advancement of Artificial Intelligence (AAAI).
 Gerevini et al. (2011) Gerevini, A., Saetti, A., and Vallati, M. (2011). PbP2: Automatic Configuration of a PortfolioBased MultiPlanner.
 Ghallab et al. (2004) Ghallab, M., Nau, D., and Traverso, P. (2004). Automated Planning: Theory and Practice. Elsevier.
 Gomes and Selman (2001) Gomes, C. P. and Selman, B. (2001). Algorithm Portfolios. Artificial Intelligence (AIJ), 126(12):43 – 62.
 Grimme et al. (2018) Grimme, C., Kerschke, P., Emmerich, M. T. M., Preuss, M., Deutz, A. H., and Trautmann, H. (2018). Sliding to the Global Optimum: How to Benefit from NonGlobal Optima in Multimodal MultiObjective Optimization. In Proceedings of the International Global Optimization Workshop (LeGO 2018). Publication status: In Press.

Hamerly and Elkan (2003)
Hamerly, G. and Elkan, C. (2003).
Learning the k in kmeans.
In Thrun, S., Saul, L. K., and Schölkopf, B., editors, Proceedings of Advances in Neural Information Processing Systems 16 (NIPS), pages 281 – 288. MIT Press.  Hansen et al. (2016) Hansen, N., Auger, A., Mersmann, O., Tušar, T., and Brockhoff, D. (2016). COCO: A Platform for Comparing Continuous Optimizers in a BlackBox Setting. arXiv preprint, abs/1603.08785v3.
 Hansen et al. (2009) Hansen, N., Finck, S., Ros, R., and Auger, A. (2009). RealParameter BlackBox Optimization Benchmarking 2009: Noiseless Functions Definitions. Technical Report RR6829, INRIA.
 Hanster and Kerschke (2017) Hanster, C. and Kerschke, P. (2017). flaccogui: Exploratory Landscape Analysis for Everyone. In Proceedings of the 19th Annual Conference on Genetic and Evolutionary Computation (GECCO) Companion, pages 1215 – 1222. ACM.
 Hastie et al. (2009) Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, 2nd edition.
 Helmert (2006) Helmert, M. (2006). The Fast Downward Planning System. Journal of Artificial Intelligence Research (JAIR), 26:191 – 246.
 Helsgaun (2000) Helsgaun, K. (2000). An Effective Implementation of the LinKernighan Traveling Salesman Heuristic. European Journal of Operational Research (EJOR), 126:106 – 130.
 Helsgaun (2009) Helsgaun, K. (2009). General kopt submoves for the LinKernighan TSP heuristic. Mathematical Programming Computation, 1(23):119 – 163.
 Hoffmann (2011) Hoffmann, J. (2011). Analyzing Search Topology Without Running Any Search: On the Connection Between Causal Graphs and h+. Journal of Artificial Intelligence Research (JAIR), 41:155 – 229.
 Hoos et al. (2014) Hoos, H. H., Lindauer, T. M., and Schaub, T. (2014). claspfolio 2: Advances in Algorithm Selection for Answer Set Programming. Theory and Practice of Logic Programming (TPLP), 14(45):569 – 585.
 Hoos et al. (2018) Hoos, H. H., Peitl, T., Slivovsky, F., and Szeider, S. (2018). PortfolioBased Algorithm Selection for Circuit QBFs. In Proceedings of the 24th International Conference on Principles and Practice of Constraint Programming (CP), Lecture Notes in Computer Science (LNCS), pages 195 – 209. Springer.
 Howe et al. (1999) Howe, A. E., Dahlman, E., Hansen, C., Scheetz, M., and von Mayrhauser, A. (1999). Exploiting Competitive Planner Performance. In Biundo, S. and Fox, M., editors, Proceedings of the Fifth European Conference on Planning (ECP), volume 1809 of Lecture Notes in Computer Science (LNCS), pages 62 – 72. Springer.
 Hsu and McIlraith (2009) Hsu, E. I. and McIlraith, S. A. (2009). VARSAT: Integrating Novel Probabilistic Inference Techniques with DPLL Search. In Kullmann, O., editor, Proceedings of the 12th International Conference on Theory and Applications of Satisfiability Testing (SAT), volume 5584 of Lecture Notes in Computer Science (LNCS), pages 377 – 390. Springer.
 Huband et al. (2006) Huband, S., Hingston, P., Barone, L., and While, L. (2006). A Review of Multiobjective Test Problems and a Scalable Test Problem Toolkit. IEEE Transactions on Evolutionary Computation (TEVC), 10(5):477 – 506.
 Huberman et al. (1997) Huberman, B. A., Lukose, R. M., and Hogg, T. (1997). An Economics Approach to Hard Computational Problems. Science, 275(5296):51 –54.
 Hutter et al. (2006) Hutter, F., Hamadi, Y., Hoos, H. H., and LeytonBrown, K. (2006). Performance Prediction and Automated Tuning of Randomized and Parametric Algorithms. In Benhamou, F., editor, Proceedings of the 12th International Conference on Principles and Practice of Constraint Programming (CP), volume 4204 of Lecture Notes in Computer Science (LNCS), pages 213 – 228. Springer.
 Hutter et al. (2013) Hutter, F., Hoos, H., and LeytonBrown, K. (2013). An Evaluation of Sequential ModelBased Optimization for Expensive Blackbox Functions. In Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation (GECCO), pages 1209 – 1216. ACM.
 Hutter et al. (2011) Hutter, F., Hoos, H. H., and LeytonBrown, K. (2011). Sequential ModelBased Optimization for General Algorithm Configuration (Extended Version). In Coello Coello, C. A., editor, Proceedings of 5th International Conference on Learning and Intelligent Optimization (LION), volume 6683, pages 507 – 523. Springer.
 Hutter et al. (2009) Hutter, F., Hoos, H. H., LeytonBrown, K., and Stützle, T. (2009). ParamILS: An Automatic Algorithm Configuration Framework. Journal of Artificial Intelligence Research (JAIR), 36(1):267 – 306.
 Hutter et al. (2014a) Hutter, F., LópezIbáñez, M., Fawcett, C., Lindauer, T. M., Hoos, H. H., LeytonBrown, K., and Stützle, T. (2014a). AClib: A Benchmark Library for Algorithm Configuration. In Pardalos, P. M., Resende, M. G. C., Vogiatzis, C., and Walteros, J. L., editors, Proceedings of the 8th International Conference on Learning and Intelligent Optimization (LION), volume 8426 of Lecture Notes in Computer Science (LNCS), pages 36 – 40. Springer.
 Hutter et al. (2014b) Hutter, F., Xu, L., Hoos, H. H., and LeytonBrown, K. (2014b). Algorithm Runtime Prediction: Methods & Evaluation. Artificial Intelligence (AIJ), 206:79 – 111.
 Huyer and Neumaier (2009) Huyer, W. and Neumaier, A. (2009). Benchmarking of MCS on the Noiseless Function Testbed. In Proceedings of the 11th Annual Conference on Genetic and Evolutionary Computation (GECCO). ACM.
 Järvisalo et al. (2012) Järvisalo, M., Le Berre, D., Roussel, O., and Simon, L. (2012). The International SAT Solver Competitions. AI Magazine, 33(1):89 – 92.
 Jones (1995) Jones, T. (1995). Evolutionary Algorithms, Fitness Landscapes and Search. PhD Thesis, University of New Mexico.
 Jones and Forrest (1995) Jones, T. and Forrest, S. (1995). Fitness Distance Correlation as a Measure of Problem Difficulty for Genetic Algorithms. In Proceedings of the 6th International Conference on Genetic Algorithms (ICGA), pages 184 – 192. Morgan Kaufmann Publishers Inc.
 Kadioglu et al. (2011) Kadioglu, S., Malitsky, Y., Sabharwal, A., Samulowitz, H., and Sellmann, M. (2011). Algorithm Selection and Scheduling. In Lee, J., editor, Proceedings of the 17th International Conference on Principles and Practice of Constraint Programming (CP), volume 6876 of Lecture Notes in Computer Science (LNCS), pages 454 – 469. Springer.
 Kadioglu et al. (2010) Kadioglu, S., Malitsky, Y., Sellmann, M., and Tierney, K. (2010). ISAC – InstanceSpecific Algorithm Configuration. In Coelho, H., Studer, R., and Wooldridge, M., editors, Proceedings of the 19th European Conference on Artificial Intelligence (ECAI), volume 215 of Frontiers in Artificial Intelligence and Applications, pages 751 – 756. IOS Press.
 Kanda et al. (2011) Kanda, J., Carvalho, A., Hruschka, E., and Soares, C. (2011). Selection of Algorithms to Solve Traveling Salesman Problems Using MetaLearning. International Journal of Hybrid Intelligent Systems, 8(3):117 – 128.
 Karatzoglou et al. (2004) Karatzoglou, A., Smola, A., Hornik, K., and Zeileis, A. (2004). kernlab – An S4 Package for Kernel Methods in R. Journal of Statistical Software (JSS), 11(9):1 – 20.
 Kauffman (1993) Kauffman, S. A. (1993). The Origins of Order: SelfOrganization and Selection in Evolution. Oxford University Press.
 Kerschke (2017a) Kerschke, P. (2017a). Automated and FeatureBased Problem Characterization and Algorithm Selection Through Machine Learning. PhD Thesis, University of Münster.
 Kerschke (2017b) Kerschke, P. (2017b). Comprehensive FeatureBased Landscape Analysis of Continuous and Constrained Optimization Problems Using the RPackage flacco. arXiv preprint, abs/1708.05258.
 Kerschke (2017c) Kerschke, P. (2017c). flacco: FeatureBased Landscape Analysis of Continuous and Constrained Optimization Problems. Rpackage version 1.6.
 Kerschke et al. (2018a) Kerschke, P., Bossek, J., and Trautmann, H. (2018a). Parameterization of StateoftheArt Performance Indicators: A Robustness Study Based on Inexact TSP Solvers. In Proceedings of the 20th Annual Conference on Genetic and Evolutionary Computation (GECCO) Companion, pages 1737 – 1744. ACM.
 Kerschke and Grimme (2017) Kerschke, P. and Grimme, C. (2017). An Expedition to Multimodal MultiObjective Optimization Landscapes. In Trautmann, H., Rudolph, G., Kathrin, K., Schütze, O., Wiecek, M., Jin, Y., and Grimme, C., editors, Proceedings of the 9th International Conference on Evolutionary MultiCriterion Optimization (EMO), pages 329 – 343. Springer.
 Kerschke et al. (2017) Kerschke, P., Kotthoff, L., Bossek, J., Hoos, H. H., and Trautmann, H. (2017). Leveraging TSP Solver Complementarity through Machine Learning. Evolutionary Computation (ECJ), pages 1 – 24.
 Kerschke et al. (2014) Kerschke, P., Preuss, M., Hernández Castellanos, C. I., Schütze, O., Sun, J.Q., Grimme, C., Rudolph, G., Bischl, B., and Trautmann, H. (2014). Cell Mapping Techniques for Exploratory Landscape Analysis. In EVOLVE – A Bridge between Probability, Set Oriented Numerics, and Evolutionary Computation V, pages 115 – 131. Springer.
 Kerschke et al. (2015) Kerschke, P., Preuss, M., Wessing, S., and Trautmann, H. (2015). Detecting Funnel Structures by Means of Exploratory Landscape Analysis. In Proceedings of the 17th Annual Conference on Genetic and Evolutionary Computation (GECCO), pages 265 – 272. ACM.
 Kerschke et al. (2016a) Kerschke, P., Preuss, M., Wessing, S., and Trautmann, H. (2016a). LowBudget Exploratory Landscape Analysis on Multiple Peaks Models. In Proceedings of the 18th Annual Conference on Genetic and Evolutionary Computation (GECCO), pages 229 – 236. ACM.
 Kerschke and Trautmann (2016) Kerschke, P. and Trautmann, H. (2016). The RPackage FLACCO for Exploratory Landscape Analysis with Applications to MultiObjective Optimization Problems. In Proceedings of the IEEE Congress on Evolutionary Computation (CEC), pages 5262 – 5269. IEEE.
 Kerschke and Trautmann (2018) Kerschke, P. and Trautmann, H. (2018). Automated Algorithm Selection on Continuous BlackBox Problems By Combining Exploratory Landscape Analysis and Machine Learning. Evolutionary Computation (ECJ), pages 1 – 28. Publication status: Accepted.
 Kerschke et al. (2016b) Kerschke, P., Wang, H., Preuss, M., Grimme, C., Deutz, A. H., Trautmann, H., and Emmerich, M. T. M. (2016b). Towards Analyzing Multimodality of Multiobjective Landscapes. In Handl, J., Hart, E., Lewis, P. R., LópezIbáñez, M., Ochoa, G., and Paechter, B., editors, Proceedings of the 14th International Conference on Parallel Problem Solving from Nature (PPSN XIV), volume 9921 of Lecture Notes in Computer Science (LNCS), pages 962 – 972. Springer.
 Kerschke et al. (2018b) Kerschke, P., Wang, H., Preuss, M., Grimme, C., Deutz, A. H., Trautmann, H., and Emmerich, M. T. M. (2018b). Search Dynamics on Multimodal MultiObjective Problems. Evolutionary Computation (ECJ), 0:1–30.
 Kotthoff (2014) Kotthoff, L. (2014). Algorithm Selection for Combinatorial Search Problems: A Survey. AI Magazine, 35(3):48 – 60.
 Kotthoff et al. (2015) Kotthoff, L., Kerschke, P., Hoos, H. H., and Trautmann, H. (2015). Improving the State of the Art in Inexact TSP Solving Using PerInstance Algorithm Selection. In Dhaenens, C., Jourdan, L., and Marmion, M.E., editors, Proceedings of the 9th International Conference on Learning and Intelligent Optimization (LION), volume 8994 of Lecture Notes in Computer Science (LNCS), pages 202 – 217. Springer.
 Kotthoff et al. (2017) Kotthoff, L., Thornton, C., Hoos, H. H., Hutter, F., and LeytonBrown, K. (2017). AutoWEKA 2.0: Automatic Model Selection and Hyperparameter Optimization in WEKA. Journal of Machine Learning Research (JMLR), 18(25):1 – 5.
 Kovárik and Málek (2012) Kovárik, O. and Málek, R. (2012). MetaLearning and MetaOptimization. Technical report, CTU Technical Report KJB2012010501 003, Prague.
 Krishnapuram et al. (2005) Krishnapuram, B., Carin, L., Figueiredo, M. A. T., and Hartemink, A. J. (2005). Sparse multinomial logistic regression: Fast algorithms and generalization bounds. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(6):957–968.
 LeytonBrown et al. (2003) LeytonBrown, K., Nudelman, E., Andrew, G., McFadden, J., and Shoham, Y. (2003). A Portfolio Approach to Algorithm Selection. In Proceedings of the Eighteenth International Joint Conference on Artificial Intelligence (IJCAI), pages 1542 – 1543. Morgan Kaufmann.
 LeytonBrown et al. (2002) LeytonBrown, K., Nudelman, E., and Shoham, Y. (2002). Learning the Empirical Hardness of Optimization Problems: The Case of Combinatorial Auctions. In Van Hentenryck, P., editor, Proceedings of the 8th International Conference on Principles and Practice of Constraint Programming (CP), volume 2470 of Lecture Notes in Computer Science (LNCS), pages 556 – 572. Springer.
 Li et al. (2013) Li, X., Engelbrecht, A. P., and Epitropakis, M. G. (2013). Benchmark Functions for CEC’2013 Special Session and Competition on Niching Methods for Multimodal Function Optimization. Technical report, RMIT University, Evolutionary Computation and Machine Learning Group, Australia.
 Lin and Kernighan (1973) Lin, S. and Kernighan, B. W. (1973). An Effective Heuristic Algorithm for the TravelingSalesman Problem. Operations Research, 21(2):498 – 516.
 Lindauer (2014) Lindauer, T. M. (2014). Algorithm Selection, Scheduling and Configuration of Boolean Constraint Solvers. PhD Thesis, University of Potsdam.
 Lindauer et al. (2016) Lindauer, T. M., Bergdoll, R.D., and Hutter, F. (2016). An Empirical Study of PerInstance Algorithm Scheduling. In Proceedings of the 10th International Conference on Learning and Intelligent Optimization (LION), volume 10079 of Lecture Notes in Computer Science (LNCS), pages 253 – 259, Ischia, Italy. Springer.
 Lindauer et al. (2015a) Lindauer, T. M., Hoos, H. H., and Hutter, F. (2015a). From Sequential Algorithm Selection to Parallel Portfolio Selection. In Dhaenens, C., Jourdan, L., and Marmion, M.E., editors, Proceedings of the 9th International Conference on Learning and Intelligent Optimization (LION), volume 8994 of Lecture Notes in Computer Science (LNCS), pages 1 – 16. Springer.
 Lindauer et al. (2015b) Lindauer, T. M., Hoos, H. H., Hutter, F., and Schaub, T. (2015b). AutoFolio: An Automatically Configured Algorithm Selector. Journal of Artificial Intelligence Research (JAIR), 53:745 – 778.
 Lindauer et al. (2017a) Lindauer, T. M., Hoos, H. H., Hutter, F., and Schaub, T. (2017a). AutoFolio: An Automatically Configured Algorithm Selector (Extended Abstract). In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pages 5025 – 5029.
 Lindauer et al. (2017b) Lindauer, T. M., van Rijn, J. N., and Kotthoff, L. (2017b). Open Algorithm Selection Challenge 2017: Setup and Scenarios. In Proceedings of Machine Learning Research, volume 79, pages 1 – 7.
 Lindauer et al. (2018) Lindauer, T. M., van Rijn, J. N., and Kotthoff, L. (2018). The Algorithm Selection Competition Series 201517. arXiv preprint, abs/1805.01214.
 LópezIbáñez et al. (2016) LópezIbáñez, M., DuboisLacoste, J., Pérez Cáceres, L., Birattari, M., and Stützle, T. (2016). The irace Package: Iterated Racing for Automatic Algorithm Configuration. Operations Research Perspectives, 3:43 – 58.
 LópezIbáñez et al. (2010) LópezIbáñez, M., Paquete, L., and Stützle, T. (2010). xploratory Analysis of Stochastic Local Search Algorithms in Biobjective Optimization. In BartzBeielstein, T., Chiarandini, M., Paquete, L., and Preuss, M., editors, Experimental Methods for the Analysis of Optimization Algorithms, pages 209 – 223. Springer.
 Loshchilov et al. (2013) Loshchilov, I., Schoenauer, M., and Sebag, M. (2013). BiPopulation CMAES Algorithms with Surrogate Models and Line Searches. In Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation (GECCO), pages 1177 – 1184. ACM.
 Lunacek and Whitley (2006) Lunacek, M. and Whitley, L. D. (2006). The Dispersion Metric and the CMA Evolution Strategy. In Proceedings of the 8th Annual Conference on Genetic and Evolutionary Computation (GECCO), pages 477 – 484. ACM.
 Mahajan et al. (2004) Mahajan, Y. S., Fu, Z., and Malik, S. (2004). Zchaff2004: An Efficient SAT Solver. In Hoos, H. H. and Mitchell, D. G., editors, Proceedings of the 7th International Conference on Theory and Applications of Satisfiability Testing (SAT), volume 3542 of Lecture Notes in Computer Science (LNCS), pages 360 – 375. Springer.
 Malan and Engelbrecht (2009) Malan, K. M. and Engelbrecht, A. P. (2009). Quantifying Ruggedness of Continuous Landscapes Using Entropy. In Proceedings of the IEEE Congress on Evolutionary Computation (CEC), pages 1440 – 1447. IEEE.
 Malan and Engelbrecht (2013) Malan, K. M. and Engelbrecht, A. P. (2013). A Survey of Techniques for Characterising Fitness Landscapes and Some Possible Ways Forward. Information Sciences (JIS), 241:148 – 163.
 Malan and Moser (2018) Malan, K. M. and Moser, I. (2018). Constraint Handling Guided by Landscape Analysis in Combinatorial and Continuous Search Spaces. Evolutionary Computation (ECJ), pages 1 – 23. (in press).
 Malan et al. (2015) Malan, K. M., Oberholzer, J. F., and Engelbrecht, A. P. (2015). Characterising Constrained Continuous Optimisation Problems. In Proceedings of the IEEE Congress on Evolutionary Computation (CEC), pages 1351 – 1358. IEEE.
 Malitsky et al. (2013) Malitsky, Y., Sabharwal, A., Samulowitz, H., and Sellmann, M. (2013). Boosting Sequential Solver Portfolios: Knowledge Sharing and Accuracy Prediction. In Nicosia, G. and Pardalos, P., editors, Proceedings of the 7th International Conference on Learning and Intelligent Optimization (LION), volume 7997 of Lecture Notes in Computer Science (LNCS), pages 153 – 167. Springer.

Mansalis et al. (2018)
Mansalis, S., Ntoutsi, E., Pelekis, N., and Theodoridis, Y. (2018).
An Evaluation of Data Stream Clustering Algorithms.
Statistical Analysis and Data Mining: The ASA Data Science Journal
, 11(4):167 – 187.  Maratea et al. (2012) Maratea, M., Pulina, L., and Ricca, F. (2012). Applying Machine Learning Techniques to ASP Solving. In Dovier, A. and Santos Costa, V., editors, Technical Communications of the 28th International Conference on Logic Programming (ICLP), volume 17, pages 37 – 48, Dagstuhl, Germany. Schloss Dagstuhl–LeibnizZentrum fuer Informatik.
 Maron and Moore (1994) Maron, O. and Moore, A. W. (1994). Hoeffding Races: Accelerating Model Selection Search for Classification and Function Approximation. In Cowan, J. D., Tesauro, G., and Alspector, J., editors, Proceedings of Advances in Neural Information Processing Systems 6 (NIPS), pages 59 – 66. Morgan Kaufmann Publishers.
 Meisel et al. (2015) Meisel, S., Grimme, C., Bossek, J., Wölck, M., Rudolph, G., and Trautmann, H. (2015). Evaluation of a MultiObjective EA on Benchmark Instances for Dynamic Routing of a Vehicle. In Proceedings of the 17th Annual Conference on Genetic and Evolutionary Computation (GECCO), pages 425 – 432, New York, NY, USA. ACM.
 Mersmann et al. (2012) Mersmann, O., Bischl, B., Bossek, J., Trautmann, H., Wagner, M., and Neumann, F. (2012). Local Search and the Traveling Salesman Problem: A FeatureBased Characterization of Problem Hardness. In Hamadi, Y. and Schoenauer, M., editors, Proceedings of the 6th International Conference on Learning and Intelligent Optimization (LION), volume 7219 of Lecture Notes in Computer Science, pages 115 – 129. Springer.
 Mersmann et al. (2011) Mersmann, O., Bischl, B., Trautmann, H., Preuss, M., Weihs, C., and Rudolph, G. (2011). Exploratory Landscape Analysis. In Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation (GECCO), pages 829 – 836. ACM.
 Mersmann et al. (2013) Mersmann, O., Bischl, B., Trautmann, H., Wagner, M., Bossek, J., and Neumann, F. (2013). A Novel FeatureBased Approach to Characterize Algorithm Performance for the Traveling Salesperson Problem. Annals of Mathematics and Artificial Intelligence, 69:151 – 182.
 Morgan and Gallagher (2015) Morgan, R. and Gallagher, M. (2015). Analysing and Characterising Optimization Problems Using Length Scale. Soft Computing, pages 1 – 18.
 Muñoz Acosta et al. (2012) Muñoz Acosta, M. A., Kirley, M., and Halgamuge, S. K. (2012). Landscape Characterization of Numerical Optimization Problems Using Biased Scattered Data. In Proceedings of the IEEE Congress on Evolutionary Computation (CEC), pages 1 – 8. IEEE.
 Muñoz Acosta et al. (2013) Muñoz Acosta, M. A., Kirley, M., and Halgamuge, S. K. (2013). The Algorithm Selection Problem on the Continuous Optimization Domain. In Computational Intelligence in Intelligent Data Analysis, volume 445 of Studies in Computational Intelligence (SCI), pages 75 – 89. Springer.
 Muñoz Acosta et al. (2015a) Muñoz Acosta, M. A., Kirley, M., and Halgamuge, S. K. (2015a). Exploratory Landscape Analysis of Continuous Space Optimization Problems Using Information Content. IEEE Transactions on Evolutionary Computation (TEVC), 19(1):74 – 87.
 Muñoz Acosta and SmithMiles (2017) Muñoz Acosta, M. A. and SmithMiles, K. A. (2017). Performance Analysis of Continuous BlackBox Optimization Algorithms via Footprints in Instance Space. Evolutionary Computation (ECJ), 25(4):529 – 554.
 Muñoz Acosta et al. (2015b) Muñoz Acosta, M. A., Sun, Y., Kirley, M., and Halgamuge, S. K. (2015b). Algorithm Selection for BlackBox Continuous Optimization Problems: A Survey on Methods and Challenges. Information Sciences (JIS), 317:224 – 245.
 Muñoz Acosta et al. (2018) Muñoz Acosta, M. A., Villanova, L., Baatar, D., and SmithMiles, K. A. (2018). Instance Spaces for Machine Learning Classification. Machine Learning, 107(1):109 – 147.
 Müller and Sbalzarini (2011) Müller, C. L. and Sbalzarini, I. F. (2011). Global Characterization of the CEC 2005 Fitness Landscapes Using FitnessDistance Analysis. In Proceedings of the European Conference on the Applications of Evolutionary Computation (EvoApplications), Lecture Notes in Computer Science (LNCS), pages 294 – 303. Springer.
 Nagata and Kobayashi (1997) Nagata, Y. and Kobayashi, S. (1997). Edge Assembly Crossover: A HighPower Genetic Algorithm for the Travelling Salesman Problem. In Bäck, T., editor, Proceedings of the 7th International Conference on Genetic Algorithms (ICGA), pages 450 – 457, San Francisco, CA, USA. Morgan Kaufmann.
 Nagata and Kobayashi (2013) Nagata, Y. and Kobayashi, S. (2013). A Powerful Genetic Algorithm Using Edge Assembly Crossover for the Traveling Salesman Problem. INFORMS Journal on Computing, 25(2):346 – 363.
 Nallaperuma et al. (2013) Nallaperuma, S., Wagner, M., Neumann, F., Bischl, B., Mersmann, O., and Trautmann, H. (2013). A FeatureBased Comparison of Local Search and the Christofides Algorithm for the Travelling Salesperson Problem. In Proceedings of the Twelfth Workshop on Foundations of Genetic Algorithms (FOGA XII), pages 147 – 160. ACM.
 Naudts et al. (1997) Naudts, B., Suys, D., and Verschoren, A. (1997). Epistasis as a Basic Concept in Formal Landscape Analysis. In Bäck, T. H. W., editor, Proceedings of the 7th International Conference on Genetic Algorithms (ICGA), pages 65 – 72. Citeseer.
 Neumann et al. (2018) Neumann, A., Gao, W., Doerr, C., Neumann, F., and Wagner, M. (2018). DiscrepancyBased Evolutionary Diversity Optimization. arXiv preprint, abs/1802.05448.
 Neumann and Poursoltan (2016) Neumann, F. and Poursoltan, S. (2016). FeatureBased Algorithm Selection for Constrained Continuous Optimisation. In Proceedings of the IEEE Congress on Evolutionary Computation (CEC), pages 1461 – 1468. IEEE.
 Nudelman et al. (2004a) Nudelman, E., LeytonBrown, K., Devkar, A., Shoham, Y., and Hoos, H. H. (2004a). SATzilla: An Algorithm Portfolio for SAT. SAT Competition 2004.
 Nudelman et al. (2004b) Nudelman, E., LeytonBrown, K., Hoos, H. H., Devkar, A., and Shoham, Y. (2004b). Understanding Random SAT: Beyond the ClausestoVariables Ratio. In Wallace, M., editor, Proceedings of the 10th International Conference on Principles and Practice of Constraint Programming (CP), volume 3258 of Lecture Notes in Computer Science (LNCS), pages 438 – 452. Springer.
 Ochoa and Veerapen (2016) Ochoa, G. and Veerapen, N. (2016). Additional Dimensions to the Study of Funnels in Combinatorial Landscapes. In Proceedings of the 18th Annual Conference on Genetic and Evolutionary Computation (GECCO), pages 373 – 380. ACM.
 Ochoa et al. (2015) Ochoa, G., Veerapen, N., Whitley, L. D., and Burke, E. K. (2015). The MultiFunnel Structure of TSP Fitness Landscapes: A Visual Exploration. In Bonnevay, S., Legrand, P., Monmarché, N., Lutton, E., and Schoenauer, M., editors, Proceedings of the 12th International Conference on Artificial Evolution (Evolution Artificielle), volume Lecture Notes in Computer Science (LNCS) of 9554, pages 1 – 13. Springer.
 OrtizBayliss et al. (2015) OrtizBayliss, J. C., TerashimaMarín, H., and ConantPablos, S. E. (2015). Lifelong Learning Selection Hyperheuristics for Constraint Satisfaction Problems. In Sidorov, G. and GaliciaHaro, S. N., editors, Proceedings of the 14th Mexican International Conference on Artificial Intelligence (MICAI): Advances in Artificial Intelligence and Soft Computing, Part I, volume 9413 of Lecture Notes in Computer Science (LNCS), pages 190 – 201. Springer.
 Pál (2013) Pál, L. (2013). Comparison of Multistart Global Optimization Algorithms on the BBOB Noiseless Testbed. In Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation (GECCO), pages 1153 – 1160. ACM.
 Pérez Cáceres et al. (2017) Pérez Cáceres, L., LópezIbáñez, Hoos, H. H., and Stützle, T. (2017). An Experimental Study of Adaptive Capping in irace. In Battiti, R., Kvasov, D. E., and Sergeyev, Y. D., editors, Proceedings of the 11th International Conference on Learning and Intelligent Optimization (LION), volume 10556 of Lecture Notes in Computer Science (LNCS), pages 235 – 250. Springer.
 Pihera and Musliu (2014) Pihera, J. and Musliu, N. (2014). Application of Machine Learning to Algorithm Selection for TSP. In Proceedings of the IEEE 26th International Conference on Tools with Artificial Intelligence (ICTAI). IEEE.
 Pitzer and Affenzeller (2012) Pitzer, E. and Affenzeller, M. (2012). A Comprehensive Survey on Fitness Landscape Analysis. In Fodor, J., Klempous, R., and Suárez Araujo, C. P., editors, Recent Advances in Intelligent Engineering Systems, Studies in Computational Intelligence, pages 161 – 191. Springer.
 Poursoltan and Neumann (2015a) Poursoltan, S. and Neumann, F. (2015a). A FeatureBased Analysis on the Impact of Set of Constraints for Constrained Differential Evolution. In Arik et al. (2015), pages 344 – 355.
 Poursoltan and Neumann (2015b) Poursoltan, S. and Neumann, F. (2015b). A FeatureBased Comparison of Evolutionary Computing Techniques for Constrained Continuous Optimisation. In Arik et al. (2015), pages 332 – 343.
 Preuss (2015) Preuss, M. (2015). Multimodal Optimization by Means of Evolutionary Algorithms. Springer.
 Pulina and Tacchella (2009) Pulina, L. and Tacchella, A. (2009). A SelfAdaptive MultiEngine Solver for Quantified Boolean Formulas. Constraints, 14(1):80 – 116.
 Reshef et al. (2011) Reshef, D. N., Reshef, Y. A., Finucane, H. K., Grossman, S. R., McVean, G., Turnbaugh, P. J., Lander, E. S., Mitzenmacher, M., and Sabeti, P. C. (2011). Detecting Novel Associations in Large Data Sets. Science, 334(6062):1518 – 1524.
 Rice (1976) Rice, J. R. (1976). The Algorithm Selection Problem. Advances in Computers, 15:65 – 118.
 Rinnooy Kan and Timmer (1987) Rinnooy Kan, A. H. G. and Timmer, G. T. (1987). Stochastic Global Optimization Methods Part II: Multi Level Methods. Mathematical Programming, 39(1):57–78.
 Rizzini et al. (2015) Rizzini, M., Fawcett, C., Vallati, M., Gerevini, A. E., and Hoos, H. H. (2015). Portfolio Methods for Optimal Planning: an Empirical Analysis. In Proceedings of the IEEE 27th International Conference on Tools with Artificial Intelligence (ICTAI), pages 494 – 501. IEEE.
 Rizzini et al. (2017) Rizzini, M., Fawcett, C., Vallati, M., Gerevini, A. E., and Hoos, H. H. (2017). Static and Dynamic Portfolio Methods for Optimal Planning: An Empirical Analysis. International Journal on Artificial Intelligence Tools, 26(01):1 – 27.
 Roberts et al. (2008) Roberts, M., Howe, A. E., Wilson, B., and desJardins, M. (2008). What Makes Planners Predictable? In Rintanen, J., Nebel, B., Beck, J. C., and Hansen, E., editors, Proceedings of the Eighteenth International Conference on Automated Planning and Scheduling (ICAPS), pages 288 – 295. Association for the Advancement of Artificial Intelligence (AAAI).
 Rochet et al. (1997) Rochet, S., Venturini, G., Slimane, M., and El Kharoubi, E. (1997). A Critical and Empirical Study of Epistasis Measures for Predicting GA Performances: A Summary. In European Conference on Artificial Evolution (AE), volume 1363 of Lecture Notes in Computer Science (LNCS), pages 275 – 285. Springer.
 Rokach (2010) Rokach, L. (2010). EnsembleBased Classifiers. Artificial Intelligence Review, 33(1 – 2):1 – 39.
 Rosé et al. (1996) Rosé, H., Ebeling, W., and Asselmeyer, T. (1996). The Density of States – A Measure of the Difficulty of Optimisation Problems. In Voigt, H.M., Ebeling, W., Rechenberg, I., and Schwefel, H.P., editors, Proceedings of the 4th International Conference on Parallel Problem Solving from Nature (PPSN IV), volume 1141 of Lecture Notes in Computer Science (LNCS), pages 208 – 217. Springer.
 Roussel (2012) Roussel, O. (2012). Description of ppfolio. Proceedings of SAT Challenge 2012: Solver and Benchmark Descriptions, page 46.
 Sanches et al. (2017a) Sanches, D., Whitley, L. D., and Tinós, R. (2017a). Building a Better Heuristic for the Traveling Salesman Problem: Combining Edge Assembly Crossover and Partition Crossover. In Proceedings of the 19th Annual Conference on Genetic and Evolutionary Computation (GECCO), pages 329 – 336. ACM.
 Sanches et al. (2017b) Sanches, D., Whitley, L. D., and Tinós, R. (2017b). Improving an Exact Solver for the Traveling Salesman Problem Using Partition Crossover. In Proceedings of the 19th Annual Conference on Genetic and Evolutionary Computation (GECCO), pages 337 – 344. ACM.
 Sander et al. (1998) Sander, J., Ester, M., Kriegel, H.P., and Xu, X. (1998). DensityBased Clustering in Spatial Databases: The Algorithm GDBSCAN and its Applications. Data Mining and Knowledge Discovery, 2(2):169 – 194.

Seo and Moon (2007)
Seo, D.I. and Moon, B.R. (2007).
An InformationTheoretic Analysis on the Interactions of Variables in Combinatorial Optimization Problems.
Evolutionary Computation (ECJ), 15(2):169 – 198.  Shirakawa and Nagao (2016) Shirakawa, S. and Nagao, T. (2016). Bag of Local Landscape Features for Fitness Landscape Analysis. Soft Computing, 20(10):3787–3802.
 Sim et al. (2015) Sim, K., Hart, E., and Paechter, B. (2015). A Lifelong Learning Hyperheuristic Method for Bin Packing. Evolutionary Computation (ECJ), 23(1):37 – 67.
 Smith et al. (2002) Smith, T., Husbands, P., and O’Shea, M. (2002). Fitness Landscapes and Evolvability. Evolutionary Computation (ECJ), 10(1):1 – 34.
 SmithMiles et al. (2010) SmithMiles, K., van Hemert, J. I., and Lim, X. Y. (2010). Understanding TSP Difficulty by Learning from Evolved Instances. In Blum, C. and Battiti, R., editors, Proceedings of the 4th International Conference on Learning and Intelligent Optimization (LION), volume 6073 of Lecture Notes in Computer Science (LNCS), pages 266 –280. Springer.

SmithMiles (2008)
SmithMiles, K. A. (2008).
Towards Insightful Algorithm Selection for Optimisation Using
MetaLearning Concepts.
In
Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN)
, pages 4118 – 4124. IEEE.  SmithMiles (2009) SmithMiles, K. A. (2009). CrossDisciplinary Perspectives on MetaLearning for Algorithm Selection. ACM Computing Surveys (CSUR), 41:1 – 25.
 SmithMiles and van Hemert (2011) SmithMiles, K. A. and van Hemert, J. I. (2011). Discovering the Suitability of Optimisation Algorithms by Learning from Evolved Instances. Annals of Mathematics and Artificial Intelligence, 61(2):87 – 104.
 Stadler (2002) Stadler, P. F. (2002). Fitness Landscapes. In Lässig, M. and Valleriani, A., editors, Biological Evolution and Statistical Physics, volume Lecture Notes in Physics (LNP) of 585, pages 183 – 204. Springer.
 Sun et al. (2014) Sun, Y., Halgamuge, S. K., Kirley, M., and Muñoz Acosta, M. A. (2014). On the Selection of Fitness Landscape Analysis Metrics for Continuous Optimization Problems. In Proceedings of the 7th International Conference on Information and Automation for Sustainability (ICIAfS), pages 1 – 6. IEEE.
 Sun et al. (2017) Sun, Y., Kirley, M., and Halgamuge, S. K. (2017). Quantifying Variable Interactions in Continuous Optimization Problems. IEEE Transactions on Evolutionary Computation (TEVC), 21(2):249 – 264.
 Tang et al. (2014) Tang, K., Peng, F., Chen, G., and Yao, X. (2014). PopulationBased Algorithm Portfolios with Automated Constituent Algorithms Selection. Information Sciences (JIS), 279:94 – 104.
 Thornton et al. (2013) Thornton, C., Hutter, F., Hoos, H. H., and LeytonBrown, K. (2013). AutoWEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms. In Dhillon, I. S., Koren, Y., Ghani, R., Senator, T. E., Bradley, P., Parekh, R., He, J., Grossman, R. L., and Uthurusamy, R., editors, Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pages 847 – 855. ACM.
 Ting (2002) Ting, K. M. (2002). An InstanceWeighting Method to Induce CostSensitive Trees. IEEE Transactions on Knowledge and Data Engineering, 14(3):659 – 665.
 Tinós et al. (2018) Tinós, R., Helsgaun, K., and Whitley, L. D. (2018). Efficient Recombination in the LinKernighanHelsgaun Traveling Salesman Heuristic. In Auger, A., da Fonseca, C. M. M., Lourenço, N., Machado, P., Paquete, L., and Whitley, L. D., editors, Proceedings of the 15th International Conference on Parallel Problem Solving from Nature (PPSN XV), volume 11101 of Lecture Notes in Computer Science (LNCS), pages 95 – 107. Springer.
 Tu and Lin (2010) Tu, H.H. and Lin, H.T. (2010). Onesided Support Vector Regression for Multiclass Costsensitive Classification. In Proceedings of the 27th International Conference on Machine Learning (ICML), pages 1095 – 1102.
 Tušar (2014) Tušar, T. (2014). Visualizing Solution Sets in Multiobjective Optimization. PhD Thesis, Jožef Stefan International Postgraduate School.
 Tušar and Filipič (2015) Tušar, T. and Filipič, B. (2015). Visualization of Pareto Front Approximations in Evolutionary Multiobjective Optimization: A Critical Review and the Prosection Method. IEEE Transactions in Evolutionary Computation (TEVC), 19(2):225 – 245.
 Tušar et al. (2016) Tušar, T., Brockhoff, D., Hansen, N., and Auger, A. (2016). COCO: The BiObjective Black Box Optimization Benchmarking (bbobbiobj) Test Suite. arXiv preprint, abs/1604.00359.
 Ugray et al. (2007) Ugray, Z., Lasdon, L., Plummer, J., Glover, F., Kelly, J., and Martí, R. (2007). Scatter Search and Local NLP Solvers: A Multistart Framework for Global Optimization. INFORMS Journal on Computing, 19(3):328 – 340.
 Ulrich et al. (2010) Ulrich, T., Bader, J., and Thiele, L. (2010). Defining and Optimizing IndicatorBased Diversity Measures in Multiobjective Search. In Schaefer, R., Cotta, C., Kołlodziej, J., and Rudolph, G., editors, Proceedings of the 11th International Conference on Parallel Problem Solving from Nature (PPSN XI), volume 6238 of Lecture Notes in Computer Science (LNCS), pages 707 – 717. Springer.
 Vallati et al. (2015) Vallati, M., Chrpa, L., Grześ, M., McCluskey, T. L., Roberts, M., and Sanner, S. (2015). The 2014 International Planning Competition: Progress and Trends. AI Magazine, 36(3):90 – 98.
 Vallati et al. (2013) Vallati, M., Chrpa, L., and Kitchin, D. (2013). An Automatic Algorithm Selection Approach for Planning. In Proceedings of the IEEE 25th International Conference on Tools with Artificial Intelligence (ICTAI), pages 1 – 8. IEEE.
 Vallati et al. (2014) Vallati, M., Chrpa, L., and Kitchin, D. (2014). ASAP: An Automatic Algorithm Selection Approach for Planning. International Journal on Artificial Intelligence Tools, 23(06):1460032.
 van Hemert (2006) van Hemert, J. I. (2006). Evolving Combinatorial Problem Instances That Are Difficult to Solve. Evolutionary Computation (ECJ), 14(4):433 – 462.
 van Rijn et al. (2013) van Rijn, J. N., Bischl, B., Torgo, L., Gao, B., Umaashankar, V., Fischer, S., Winter, P., Wiswedel, B., Berthold, M. R., and Vanschoren, J. (2013). OpenML: A Collaborative Science Platform. In Blockeel, H., Kersting, K., Nijssen, S., and Železný, F., editors, Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML/PKDD), volume 8190 of Lecture Notes in Computer Science (LNCS), pages 645 – 649. Springer.
 van Rijn et al. (2014) van Rijn, J. N., Holmes, G., Pfahringer, B., and Vanschoren, J. (2014). Algorithm Selection on Data Streams. In Džeroski, S., Panov, P., Kocev, D., and Todorovski, L., editors, Proceedings of the 17th International Conference on Discovery Science (DS), volume 8777 of Lecture Notes in Computer Science (LNCS), pages 325 – 336. Springer.
 van Rijn et al. (2018) van Rijn, J. N., Holmes, G., Pfahringer, B., and Vanschoren, J. (2018). The Online Performance Estimation Framework: Heterogeneous Ensemble Learning for Data Streams. Machine Learning, 107(1):149 – 176.
 van Rijn et al. (2017) van Rijn, S., Wang, H., van Stein, B., and Bäck, T. (2017). Algorithm Configuration Data Mining For CMA Evolution Strategies. In Proceedings of the 19th Annual Conference on Genetic and Evolutionary Computation (GECCO), pages 737 – 744. ACM.
 van Veldhuizen (1999) van Veldhuizen, D. A. (1999). Multiobjective Evolutionary Algorithms: Classifications, Analyzes, and New Innovations. PhD Thesis, Faculty of the Graduate School of Engineering of the Air Force Institute of Technology, Air University.

Vapnik (1995)
Vapnik, V. N. (1995).
The Nature of Statistical Learning Theory
. Springer.  Vassilev et al. (2000) Vassilev, V. K., Fogarty, T. C., and Miller, J. F. (2000). Information Characteristics and the Structure of Landscapes. Evolutionary Computation (ECJ), 8(1):31 – 60.
 Wagner et al. (2017) Wagner, M., Lindauer, T. M., Mısır, M., Nallaperuma, S., and Hutter, F. (2017). A Case Study of Algorithm Selection for the Traveling Thief Problem. Journal of Heuristics, pages 1 – 26.
 Wessing (2015) Wessing, S. (2015). TwoStage Methods for Multimodal Optimization. PhD Thesis, Technische Universität Dortmund.
 Wessing (2016) Wessing, S. (2016). optproblems: Infrastructure to define optimization problems and some test problems for blackbox optimization. Pythonpackage version 0.6.
 Witten et al. (2016) Witten, I. H., Frank, E., Hall, M. A., and Pal, C. J. (2016). Data Mining: Practical Machine Learning Tools and Techniques. Morgan Kaufmann.
 Wolpert and Macready (1995) Wolpert, D. H. and Macready, W. G. (1995). No Free Lunch Theorems for Search. Technical report, Santa Fe Institute, Santa Fe, NM, USA. SFITR9502010.
 Wolpert and Macready (1997) Wolpert, D. H. and Macready, W. G. (1997). No Free Lunch Theorems for Optimization. IEEE Transactions on Evolutionary Computation (TEVC), 1(1):67 – 82.
 Wright (1932) Wright, S. (1932). The Roles of Mutation, Inbreeding, Crossbreeding and Selection in Evolution. In Proceedings of the 6th International Congress of Genetics, volume 1, pages 356 – 366.
 Xie and Liu (2009) Xie, X.F. and Liu, J. (2009). Multiagent Optimization System for Solving the Traveling Salesman Problem (TSP). IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, 39(2):489 – 502.
 Xu et al. (2007) Xu, L., Hoos, H. H., and LeytonBrown, K. (2007). Hierarchical Hardness Models for SAT. In Bessière, C., editor, Proceedings of the 13th International Conference on Principles and Practice of Constraint Programming (CP), volume 4741 of Lecture Notes in Computer Science (LNCS), pages 696 – 711. Springer.
 Xu et al. (2008) Xu, L., Hutter, F., Hoos, H. H., and LeytonBrown, K. (2008). SATzilla: PortfolioBased Algorithm Selection for SAT. Journal of Artificial Intelligence Resesearch (JAIR), 32:565 – 606.
 Xu et al. (2012) Xu, L., Hutter, F., Hoos, H. H., and LeytonBrown, K. (2012). Evaluating Component Solver Contributions to PortfolioBased Algorithm Selectors. In Cimatti, A. and Sebastiani, R., editors, Proceedings of the 15th International Conference on Theory and Applications of Satisfiability Testing (SAT), volume 7317 of Lecture Notes in Computer Science (LNCS), pages 228 – 241. Springer.
 Yuen and Zhang (2015) Yuen, S. Y. and Zhang, X. (2015). On composing an algorithm portfolio. Memetic Computing, 7(3):203 – 214.
 Zhang et al. (2008) Zhang, Q., Zhou, A., Zhao, S., Suganthan, P. N., Liu, W., and Tiwari, S. (2008). Multiobjective Optimization Test Instances for the CEC 2009 Special Session and Competition. Technical report, University of Essex, Colchester, UK and Nanyang Technological University, Singapore. Special Session on Performance Assessment of MultiObjective Optimization Algorithms.
 Zitzler et al. (2000) Zitzler, E., Deb, K., and Thiele, L. (2000). Comparison of Multiobjective Evolutionary Algorithms: Empirical Results. Evolutionary Computation (ECJ), (2):173 – 195.