1 Introduction
It has been widely observed in many problem domains [Xu et al.2010, Tang et al.2014] that there is no universal optimal solver dominating all other solvers on all problem instances. Instead, different solvers perform well on different problem instances. Thus a natural idea is to combine those complementary solvers together to achieve a better overall performance. Typical examples include algorithm selection methods [Rice1976, Xu et al.2008, Kotthoff2016] which try to select the best solver for every single problem instance before solving it, and adaptive solvers such as adaptive parameter control [Karafotias et al.2015], reactive search [Battiti et al.2008]
and hyperheuristics
[Burke et al.2013] which seek to dynamically determine the best solver setting while solving a problem instance. In principle, all these methods need to involve some mechanisms (e.g., selection or scheduling) to appropriately allocate computational resource to different solvers.Recently parallel portfolios [Gomes and Selman2001, Tang et al.2014, Lindauer et al.2017] as another paradigm of simultaneously utilizing several sequential solvers, have attracted more and more research interest. While solving a problem instance, parallel portfolios run all the component solvers in parallel until the first of them solves it; thus the performance of a parallel portfolio is always the best performance achieved by its component solvers. Different from algorithm selection and adaptive solvers, parallel portfolios do not necessarily require any extra resource allocation since each involved component solver is simply assigned with the same amount of resource. Moreover, the rapid growth of parallelism in computational power [Gepner and Kowalik2006] makes such parallel solving strategy more and more critical for solving computationally hard problems. However, the manual construction of parallel portfolios is nontrivial. Specifically, identifying (or designing) a set of relatively uncorrelated sequential solvers which are complementary to each other still requires relatively significant domain knowledge [Xu et al.2010].
Recently [Lindauer et al.2017] proposed using automatic construction of parallel portfolios (ACPP) as a first step towards tackling parallel portfolio construction. The goal of ACPP is to automatically construct a parallel portfolio based on the rich design space induced by a highly parametrized sequential solver or a set of them. More formally, let and denote the configuration space of the parameterized solvers and the given set of problem instances, respectively. The parallel portfolio with component solvers could be denoted as a tuple, i.e., , in which is an individual configuration and represents the th component solver of . The goal is to find from , such that the performance of on according to a given metric is optimized.
There are three key ACPP methods, namely GLOBAL, and CLUSTERING, in which GLOBAL and are both proposed by [Lindauer et al.2017] while CLUSTERING is adapted by [Lindauer et al.2017] from [Kadioglu et al.2010] for comparison ^{1}^{1}1Although is an automatic portfolio construction method for algorithm selection, its basic idea is also applicable to ACPP. See [Lindauer et al.2017] for more details.. Generally, these methods can be divided into two categories. GLOBAL belongs to the first category, which considers ACPP an algorithm configuration problem by directly treating as a parameterized solver. By this means powerful automatic algorithm configurators such as ParamILS [Hutter et al.2009], GGA [Ansótegui et al.2009], irace [Ansótegui et al.2009] and SMAC [Hutter et al.2011] could be directly applied to configure (GLOBAL uses SMAC in [Lindauer et al.2017]). The key issue of GLOBAL is that its scalability is limited since the size of the configuration space of , i.e., , increases exponentially with the number of the component solvers, i.e., . The other two methods, and CLUSTERING, solve the ACPP problem from the perspective of instance grouping. That is, they explicitly or implicitly promote different component solvers of to handle different subsets of problem instances, with the goal that the resulting component solvers would be complementary to each other. More specifically, starting from an empty portfolio, proceeds iteratively and in the th iteration it uses an algorithm configurator (also SMAC) to configure to add to the current portfolio, i.e., , such that the performance of the resulting portfolio, i.e., , is optimized. In other words, in each iteration intrinsically aims to find a solver that can improve the current portfolio to the best extent. Since the greatest chance of the current portfolio getting improved is on those problem instances which cannot be solved satisfactorily, thus while configuring the th component solver , the configurator would actually promote to handle those intractable instances to the current portfolio . Compared to , CLUSTERING adopts a more explicit way to group instances. It clusters the problem instances in a given (normalized) instance feature space and then independently configures (using SMAC) a component solver on each instance cluster. The advantage of PARHDYRA and CLUSTERING is that they keep the size of the configuration space involved in each algorithm configuration task as .
The main issue of is that its intrinsic greedy mechanism may cause stagnation in local optima. To alleviate this problem, [Lindauer et al.2017] makes a modification to by allowing simultaneously configuring several component solvers in each iteration. The resulting method is named , where is the number of the component solvers configured in each iteration. and GLOBAL could be both seen as special cases of with and respectively. It is conceivable that the choice of is very important for since the tendency to stagnate in local optima would increase as gets smaller, while the size of the configuration space involved in each configuration task in , i.e., , would grow exponentially as gets larger. However, in general the best value of may vary across different scenarios, and for a specific scenario it is very hard to determine a good choice of in advance.
For methods based on explicit instance grouping such as CLUSTERING, obviously the quality of the instance grouping is crucial. A good instance grouping should meet at least one requirement: The instances that are grouped together should be similar in the sense that in the configuration space there exist same good configurations for them. CLUSTERING uses the distances in the normalized feature space to characterize such similarity. Although the problem features have been proved very useful for modeling the relationship between problem instances and solvers (e.g., algorithm selection), some practical issues still exist while applying CLUSTERING to ACPP. Specifically, the used instance features as well as the normalization of the features would greatly influence the final clustering results. However, determining appropriate choices of them is very hard since accurate assessment of the cluster quality relies on completely constructed portfolios.
In this paper, we investigate further solving the ACPP problem based on explicit instance grouping. Specifically, we propose a new ACPP method named parallel configuration with instance transfer (PCIT) ^{2}^{2}2Here “parallel configuration” means that the configuration processes of the component solvers are independent of each other and thus could be conducted in parallel., which explicitly divides the instances into different subsets and configures a component solver on each subset. The most novel feature of PCIT is its dynamic instance transfer mechanism. Unlike CLUSTERING which determines the instance grouping in advance and then keeps it fixed through the whole process, during portfolio construction PCIT would dynamically adjust the instance grouping by transferring instances between different subsets. The instance transfer is conducted with the goal that the instances which share the same highquality configurations (in the configuration space ) would be grouped together, such that the complementarity between the component solvers configured on different subsets would be favourably enhanced. The experimental results showed that the parallel portfolios constructed by PCIT could achieve consistently superior performances to the ones output by the existing ACPP methods, and could even rival the stateoftheart handdesigned parallel solvers.
2 Proposed Method
The basic idea of PCIT is simple. Although it is hard to obtain an appropriate instance grouping at one stroke, it is possible to gradually improve an instance grouping. PCIT adopts a random splitting strategy to obtain an initial grouping; that is, the instances are evenly and randomly divided into subsets. It is conceivable that the quality of random instance grouping is not guaranteed at all since there is no guidance involved in the grouping procedure. Consider a simple example where instance set , configuration space , shares the highquality configuration and shares the highquality configuration . Obviously the appropriate grouping for this example is , which would lead algorithm configurator to output and on the first and the second subset respectively, thus producing the optimal portfolio . Random splitting strategy may fail on this example if it happens to split as or , which could cause algorithm configurator to output the same component solver i.e., or , on both subsets.
The key point here is that if the problem instances grouped together do not share the same highquality configurations, then the cooperation between the component solvers configured on these subsets would be much affected, thus limiting the quality of the final output parallel portfolio. To handle this issue, PCIT employs an instance transfer mechanism to improve the instance grouping during the construction process by transferring instances between different subsets. More specifically, as the configuration process of a component solver on a subset proceeds, if the algorithm configurator cannot manage to find a common highperformance configuration for every instance in the subset but only some of them, then it can be inferred that these intractable instances may correspond to different highquality configurations (in the configuration space ) from others. It is therefore better to transfer these instances to other subsets that are more suitable to them.
PCIT conducts the instance transfer with the help of incumbent configurations (i.e., the best configurations found by the algorithm configurator). In each subset, the instances which cannot be solved satisfactorily by the corresponding incumbent are identified as the ones that need to be transferred, and the target subset of each transferred instance is determined according to how well the incumbent on the candidate subset could perform on the instance. In essence, the incumbent on a subset can be seen as a common special characteristic of those “similar” instances (in the sense they share the same highquality configurations) within the subset, and PCIT uses it to identify those “dissimilar” instances and find better subsets for them. In each subset, the performance of the incumbent on each instance could be obtained from the rundata collected from the configuration process. However, while determining the target subsets for the transferred instances, how well the incumbents on the candidate subsets would perform on the transferred instances are unknown. One way to obtain these performances is to actually test these incumbents on the transferred instances, which however would introduce considerable additional computational costs. To avoid this, PCIT builds empirical performance models (EPM) [Hutter et al.2014] based on the collected rundata to predict these performances.
2.1 Algorithm Framework
The pseudocode of PCIT is given in Algorithm 1. The main difference between PCIT and the existing methods (e.g., GLOBAL and CLUSTERING) is that in PCIT the portfolio construction process is divided into (we always set in this paper) sequential phases (Lines 313 in Algorithm 1). The first phases serve as adjustment phases, in each of which the instance grouping is adjusted (Line 12) once the configuration procedures for all component solvers (Lines 911) finish. The last phase is the construction phase in which the component solvers of the final portfolio are configured on the obtained subsets with a large amount of time. In fact, the time consumed for the configuration processes in the last phase amounts to the sum of the time consumed for the configuration processes in the first phases (Lines 48). One thing which is not detailed in Algorithm 1 for brevity is that, on each subset, to keep the continuity of the configuration processes across successive phases, the incumbent configuration obtained in the previous phase is always used to initialize the configuration procedure in the next phase.
Another important difference between PCIT and the existing methods lies in the way of obtaining reliable outputs. For existing methods, the uncertainty of the portfolio construction results mainly comes from the randomness of the output of the algorithm configurator (especially when the given parametrized solvers are not deterministic). Thus for each specific algorithm configuration task, typically they conduct multiple independent runs of the algorithm configurator (with different random seeds), and then validate the configurations produced by these runs to determine the output one. For PCIT, in addition to the randomness mentioned above, a greater source of uncertainty is the randomness of the initial instance grouping results. One way to handle both of them is to perform multiple runs of portfolio construction (with different initial instance groupings), and in each construction process the algorithm configurator is also run for multiple times for each configuration task. In this paper, to keep the design simple, we only allow repeated runs of portfolio construction and rely on the validation to ensure the reliability of the final output (Lines 1618 in Algorithm 1).
PCIT can be easily performed in parallel. First, different portfolio construction runs (Lines 115 in Algorithm 1) can be executed in parallel, and second, during each construction run the configuration processes for different component solvers (Lines 911 in Algorithm 1) can also be executed in parallel.
2.2 Instance Transfer
As shown in Algorithm 2, the instance transfer procedure first builds an empirical performance model (EPM) based on the rundata collected from all the previous algorithm configuration processes (Line 1). More specifically, the rundata is actually records of runs of different solver configurations on different instances, and each run can be represented by a 3tuple, i.e., . The exact implementation of EPM here is the same as the one in SMAC [Hutter et al.2011]
, which is a random forest that takes as input a solver configuration
and a problem instance(represented by a feature vector), and predicts performance of
on . The performances of the incumbent configuration on instances in each subset are obtained by querying the corresponding runs in rundata ^{3}^{3}3The average performance is used if there are several such runs. In case that there is no such run in rundata, which means through the configuration process the incumbent configuration has not been tested on the instance yet, the instance will be excluded from the whole transfer process. (Line 2). After collecting all of them, the median value is used to identify the instances which will be transferred (without loss of generality, we assume a smaller value is better for ) (Line 3). Then these instances are examined one by one in a random order (Lines 722), for each examined instance the target subset is determined according to three rules (Line 13): 1) Both the source subset and the target subset will not violate the constraints on the subset size after the instance is transferred; 2) The predicted performance on the instance is not worse on the target subset; 3) The target subset is the one with the best predicted performance among the ones satisfying 1) and 2). The subset size constraints, i.e., the lower bound and the upper bound in Algorithm 2, are set to prevent the occurrence of too large or too small subsets. In this paper and are set as , respectively. Since the sizes of the subsets keep changing during the instance transfer process, there is a possibility that an instance, which was examined earlier and at that time no target subset satisfying the above conditions was found, has a satisfactory target subset later. To handle this situation, instances which are not successfully transferred will be examined again in the next round (Line 26), and the whole procedure will be terminated (Lines 2325) if there is no instance which needs to be transferred, or there is no successful transfer in a round (Lines 722).2.3 Computational Costs
The computational costs of ACPP methods are mainly composed of two parts: the costs of configuration processes and the costs of validation. For PCIT, the total CPU time consumed is (the small overhead introduced by instance transfer in PCIT is ignored here). Similarly, for GLOBAL and CLUSTERING, it is , where is the number of independent runs of algorithm configurator (for each configuration task). For , the consumed CPU time is , where and refer in particular to the configuration time budget and the validation time budget used in (See [Lindauer et al.2017] for more details).
3 Empirical Study
We conducted experiments on two widely studied domains, SAT and TSP. Specifically, we used our method to build parallel portfolios based on a training set, and then compared them with the ones constructed by the existing methods, on an unseen test set.
3.1 Experimental Setup
3.1.1 Portfolio Size and Performance Metric
We set the number of component solvers to 8 (same as [Lindauer et al.2017]), since 8core (and 8thread) machines are widely available now. The optimization goal considered here is to minimize the time required by a solver to solve the problem (for SAT) or to find the optimum of the problem (for TSP). In particular, we set the performance metric to Penalized Average Runtime–10 (PAR10) [Hutter et al.2009], which counts each timeout as 10 times the given cutoff time. The optimal solutions for TSP instances were obtained using Concorde [Applegate et al.2006], an exact TSP solver.
3.1.2 Scenarios
For each problem domain we considered constructing portfolios based on a single parameterized solver and based on multiple parameterized solvers, resulting in four different scenarios. For brevity, we use SAT/TSPSingle/Multi to denote these scenarios. Table 1 summarizes the used instance sets, cutoff time, and base parameterized solvers in each scenario. Except in SATMulti we reused the settings from [Lindauer et al.2017], in the other three scenarios we all used new settings which had never been considered before in the literature of ACPP. We especially note that this was the first time the ACPP methods were applied to TSP. Settings in SATMulti are the same as the ones in [Lindauer et al.2017]: 1) Instance set obtained from the application track of the SAT’12 Challenge were randomly and evenly split into a training set and a test set, and to ensure the computational costs for portfolio construction would not be prohibitively large, the cutoff time used in training (180s) was smaller than the one used in testing (900s, same as the SAT’12 challenge); 2) The parameterized solvers in SATMulti (the configuration space contains 150 parameters in total, including a toplevel parameter used to select a base solver) were the 8 sequential solvers considered by [Wotzlaw et al.2012] when designing pfolioUZK, the gold medal winning solver in the parallel track of the SAT’12 Challenge. In SATSingle, we chose instances from the benchmark used in the agile track of the SAT’16 Competition for its moderate cutoff time (60s). Specifically, we randomly selected 2000 instances from the original benchmark (containing 5000 instances) and divided them evenly for training and testing. We chose Riss6 [Manthey et al.2016], the gold medal winning solver of this track, as the base solver. Since Riss6 exposes a large number of parameters, we selected 135 parameters from them to be tunable while leaving others as default. For TSPSingle and TSPMulti we used a same instance set. Specifically, we used the and the generators from the 8th DIMACS Implementation Challenge to generate 1000 “uniform” instances (in which the cities are randomly distributed) and 1000 “clustering” instances (in which the cities are distributed around different central points). The problem sizes of all these generated instances are within . Once again, we divided them evenly for training and testing. The base solver used in TSPSingle was LKH version 2.0.7 [Helsgaun2000] (with 35 parameters), one of the stateoftheart inexact solver for TSP. In TSPMulti, in addition to LKH, we included another two powerful TSP solvers, GAEAX version 1.0 [Nagata and Kobayashi2013] (with 2 parameters) and CLK [Applegate et al.2003] (with 4 parameters), as the base solvers, resulting in a configuration space containing 43 parameters (including a toplevel parameter used to select a base solver).
Instance Set  Cutoff Time  Base Solvers  

SATSingle  From the SAT’16 Competition Agile Track, 2000 instances (1000 for training, 1000 for testing)  60s  Riss6 [Manthey et al.2016], 
SATMulti  From the SAT’12 Challenge Application Track, 600 instances (300 for training, 300 for testing)  180s(900s)  8 solvers considered by [Wotzlaw et al.2012], 
TSPSingle  Same as TSPMulti  20s  LKH [Helsgaun2000], 
TSPMulti  2000 instances containing 1000 “uniform” ones and 1000 “clustering” ones generated using the generators from the DIMACS TSP Challenge (1000 for training, 1000 for testing)  20s  LKH [Helsgaun2000], CLK [Applegate et al.2003] and GAEAX [Nagata and Kobayashi2013], 
SATSingle  SATMulti  TSPSingle  TSPMulti  

M_group  (36,4,3200)  (80,4,6720)  (16,2,1440)  (24,2,2080) 
(6,4,3600)  (15,4,6840)  (3,2,1800)  (4,2,2160)  
(12,4,3200)  (30,4,6800)  (6,2,1600)  (8,2,2000)  
(24,4,3360)  (60,4,7680)  (12,2,1680)  (16,2,2160) 
3.1.3 Competitors and Time Budgets
Besides PCIT, we implemented GLOBAL, (with b=1,2,4), and CLUSTERING (with normalization options including linear normalization, standard normalization and no normalization), as described in [Lindauer et al.2017] for comparison. For all considered ACPP methods here, SMAC version 2.10.03 [Hutter et al.2011] was used as the algorithm configurator. Since the performance of SMAC could be often improved when used with the instance features, we gave SMAC access to the 126 SAT features used in [Hutter et al.2011], and the 114 TSP features used in [Kotthoff et al.2015]. The same features were also used by PCIT (for transferring instances) and CLUSTERING (for clustering instances). To make the comparisons fair, the total CPU time consumed by each method was kept almost the same. The detailed setting of time budget for each method is given in Table 2. To validate whether the instance transfer in PCIT is useful, we included another method, named PCRS (parallel configuration with random splitting), in the comparison. PCRS differs from PCIT in that it directly configures the final portfolios on the initial random instance grouping and involves no instance transfer. The time budgets for PCRS were the same as PCIT.
SATSingle  SATMulti  TSPSingle  TSPMulti  

#TOS  PAR10  PAR1  #TOS  PAR10  PAR1  #TOS  PAR10  PAR1  #TOS  PAR10  PAR1  
Baseline  383  238  31  71  2275  358  565  118  16  455  99  17 
PCRS  234  152  26  44  1435  247  110  31  11  105  30  11 
PCIT  181  119  21  35  1164  219  87  24  8  86  24  9 
GLOBAL  230  149  25  46  1495  253  224  53  13  150  41  14 
235 b=4  151  24  40 b=1  1326  246  107 b=1  29  10  85 b=2  24  9  
CLUSTERING  227 None  146  23  43 None  1415  254  121 Linear  31  9  99 Linear  28  10 
3.1.4 Baselines
For each scenario, we identified a sequential solver as the baseline by using SMAC to configure on the training set and the configuration space of the scenario.
3.1.5 Experimental Environment
All the experiments were conducted on a cluster of 5 Intel Xeon machines with 60 GB RAM and 6 cores each (2.20 GHz, 15 MB Cache), running Centos 7.5.
3.2 Results
We tested each solver (including the ACPP portfolios and the baseline) by running it on each test instance for 3 times, and reported the median performance. The obtained number of timeouts (#TOS), PAR10 and PAR1 are presented in Table 3. For CLUSTERING and , we always reported the best performance achieved by their different implementations. To determine whether the performance differences between these solvers were significant, we performed a permutation test (with 100000 permutations and significance level ) to the (0/1) timeout scores, the PAR10 scores and the PAR1 scores. Overall the portfolios constructed by PCIT achieved the best performances in Table 3. In SATSingle, SATMulti and TSPSingle, it achieved significantly and substantially better performances than all the other solvers. Although in TSPMulti, the portfolio constructed by obtained slightly better results than the one constructed by PCIT (however the performance difference is insignificant), as aforementioned, the appropriate value of varied across different scenarios (as shown in Table 3) and for a specific scenario it was actually unknown in advance (in TSPMulti it was 2). Similarly, as shown in Table 3, the best normalization strategy for CLUSTERING also varied across different scenarios. Compared to the portfolios constructed by PCRS, the ones constructed by PCIT consistently obtain much better results, which verified the effectiveness of the instance transfer mechanism of PCIT. Finally, all the ACPP methods here could build portfolios that obtained much better results than the baselines, indicating the great benefit by combining complementary configurations obtained from a rich design space.
To further evaluate the portfolios constructed by PCIT, we compared them with the stateoftheart manually designed parallel solvers. Specifically, we considered the ones constructed for SAT. We chose Priss6 [Manthey et al.2016] to compare with the one constructed in SATSingle, since Priss6 is the official parallel version of Riss6 (the base solver in SATSingle). For the same reason, we chose PfolioUZK [Wotzlaw et al.2012] (the gold medal winning solver of the parallel track of the SAT’12 Challenge) to compare with the one constructed in SATMulti. Finally, we chose Plingeling (version bbc) [Biere2016], the gold medal winning solver of the parallel track of the SAT’16 Competition, to compare with both. Note that all the manually designed solvers considered here have implemented far more advanced solving strategies (e.g., clause sharing) than only independently running component solvers in parallel. In the experiments the default settings of these solvers were used and the same statistical tests as before were conducted. As shown in Table 4, on SATSingle test set, the portfolio constructed by PCIT achieved much better results than others. This may be because the parallel solvers considered here are not designed for the type of these instances (obtained from the SAT’16 Competition Agile track, which is for simple fast SAT solvers with low overhead), which on the other hand demonstrates the wide applicability of ACPP methods. It is impressive that, on SATMulti test set, the portfolio constructed by PCIT (regardless of its simple solving strategy) obtained slightly better results than pfolioUZK, and could reach the performance level of the more stateoftheart Plingeling. Such results imply that the portfolios constructed by PCIT may be a good staring point for designing more powerful parallel solvers.
SATSingle  SATMulti  

#TOS  PAR10  PAR1  #TOS  PAR10  PAR1  
PCIT  181  119  21  35  1164  219 
Priss6  225  146  25       
PfolioUZK        36  1185  213 
Plineglingbbc  452  276  32  33  1090  199 
4 Conclusion
In this paper we proposed a novel ACPP method which utilized an instance transfer mechanism to improve the quality of the instance grouping. The experimental results verified the effectiveness of the proposed method. Directions of future work include extending the proposed method to use parallel solvers as base solvers, and investigating solving ACPP from the perspective of subset selection.
References

[Ansótegui et al.2009]
Carlos Ansótegui, Meinolf Sellmann, and Kevin Tierney.
A genderbased genetic algorithm for the automatic configuration of algorithms.
In Proceedings of the 15th International Conference on Principles and Practice of Constraint Programming, CP’2009, Lisbon, Portugal, September 2009.  [Applegate et al.2003] David Applegate, William Cook, and André Rohe. Chained LinKernighan for large traveling salesman problems. INFORMS Journal on Computing, 15(1):82–92, 2003.
 [Applegate et al.2006] David Applegate, Ribert Bixby, Vasek Chvatal, and William Cook. Concorde tsp solver, 2006.
 [Battiti et al.2008] Roberto Battiti, Mauro Brunato, and Franco Mascia. Reactive search and intelligent optimization, volume 45. Springer Science & Business Media, 2008.
 [Biere2016] Armin Biere. Splatz, lingeling, plingeling, treengeling, yalsat entering the sat competition 2016. Proceedings of SAT Competition, pages 44–45, 2016.
 [Burke et al.2013] Edmund K Burke, Michel Gendreau, Matthew Hyde, Graham Kendall, Gabriela Ochoa, Ender Özcan, and Rong Qu. Hyperheuristics: A survey of the state of the art. Journal of the Operational Research Society, 64(12):1695–1724, 2013.
 [Gepner and Kowalik2006] Pawel Gepner and Michal Filip Kowalik. Multicore processors: New way to achieve high system performance. In Fifth International Conference on Parallel Computing in Electrical Engineering, PARELEC’2006, Bialystok, Poland, September 2006.
 [Gomes and Selman2001] Carla P Gomes and Bart Selman. Algorithm portfolios. Artificial Intelligence, 126(12):43–62, 2001.
 [Helsgaun2000] Keld Helsgaun. An effective implementation of the LinKernighan traveling salesman heuristic. European Journal of Operational Research, 126(1):106–130, 2000.
 [Hutter et al.2009] Frank Hutter, Holger H. Hoos, Kevin LeytonBrown, and Thomas Stützle. Paramils: An automatic algorithm configuration framework. Journal of Artificial Intelligence Research, 36(1):267–306, 2009.
 [Hutter et al.2011] Frank Hutter, Holger Hoos, and Kevin LeytonBrown. Sequential modelbased optimization for general algorithm configuration. In Learning and Intelligent Optimization  5th International Conference, LION’5, Rome, Italy, January 2011.
 [Hutter et al.2014] Frank Hutter, Lin Xu, Holger H Hoos, and Kevin LeytonBrown. Algorithm runtime prediction: Methods & evaluation. Artificial Intelligence, 206:79–111, 2014.
 [Kadioglu et al.2010] Serdar Kadioglu, Yuri Malitsky, Meinolf Sellmann, and Kevin Tierney. ISAC  instancespecific algorithm configuration. In Proceedings of the 19th European Conference on Artificial Intelligence, ECAI’2010, pages 751–756, Lisbon, Portugal, August 2010.

[Karafotias et al.2015]
Giorgos Karafotias, Mark Hoogendoorn, and Ágoston E Eiben.
Parameter control in evolutionary algorithms: Trends and challenges.
IEEE Transactions on Evolutionary Computation
, 19(2):167–187, 2015.  [Kotthoff et al.2015] Lars Kotthoff, Pascal Kerschke, Holger Hoos, and Heike Trautmann. Improving the state of the art in inexact TSP solving using perinstance algorithm selection. In Learning and Intelligent Optimization  9th International Conference, LION’9, Lille, France, January 2015.
 [Kotthoff2016] Lars Kotthoff. Algorithm selection for combinatorial search problems: A survey. In Data Mining and Constraint Programming  Foundations of a CrossDisciplinary Approach, pages 149–190. Springer, 2016.
 [Lindauer et al.2017] Marius Lindauer, Holger Hoos, Kevin LeytonBrown, and Torsten Schaub. Automatic construction of parallel portfolios via algorithm configuration. Artificial Intelligence, 244:272–290, 2017.
 [Manthey et al.2016] Norbert Manthey, Aaron Stephan, and Elias Werner. Riss 6 solver and derivatives. Proceedings of SAT Competition, page 56, 2016.
 [Nagata and Kobayashi2013] Yuichi Nagata and Shigenobu Kobayashi. A powerful genetic algorithm using edge assembly crossover for the traveling salesman problem. INFORMS Journal on Computing, 25(2):346–363, 2013.
 [Rice1976] John R. Rice. The algorithm selection problem. Advances in Computers, 15:65–118, 1976.
 [Tang et al.2014] Ke Tang, Fei Peng, Guoliang Chen, and Xin Yao. Populationbased algorithm portfolios with automated constituent algorithms selection. Information Sciences, 279:94–104, 2014.
 [Wotzlaw et al.2012] Andreas Wotzlaw, Alexander van der Grinten, Ewald Speckenmeyer, and Stefan Porschen. pfolioUZK: Solver description. Proceedings of SAT Challenge, page 45, 2012.
 [Xu et al.2008] Lin Xu, Frank Hutter, Holger Hoos, and Kevin LeytonBrown. Satzilla: Portfoliobased algorithm selection for SAT. Journal of Artificial Intelligence Research, 32:565–606, 2008.
 [Xu et al.2010] Lin Xu, Holger Hoos, and Kevin LeytonBrown. Hydra: Automatically configuring algorithms for portfoliobased selection. In Proceedings of the TwentyFourth AAAI Conference on Artificial Intelligence, AAAI’2010, Atlanta, Georgia, July 2010.