DeepAI AI Chat
Log In Sign Up

Benchmarking Feature-based Algorithm Selection Systems for Black-box Numerical Optimization

by   Ryoji Tanabe, et al.

Feature-based algorithm selection aims to automatically find the best one from a portfolio of optimization algorithms on an unseen problem based on its landscape features. Feature-based algorithm selection has recently received attention in the research field of black-box numerical optimization. However, algorithm selection for black-box optimization has been poorly understood. Most previous studies have focused only on whether an algorithm selection system can outperform the single-best solver in a portfolio. In addition, a benchmarking methodology for algorithm selection systems has not been well investigated in the literature. In this context, this paper analyzes algorithm selection systems on the 24 noiseless black-box optimization benchmarking functions. First, we demonstrate that the successful performance 1 measure is more reliable than the expected runtime measure for benchmarking algorithm selection systems. Then, we examine the influence of randomness on the performance of algorithm selection systems. We also show that the performance of algorithm selection systems can be significantly improved by using a pre-solver. We point out that the difficulty of outperforming the single-best solver depends on algorithm portfolios, cross-validation methods, and dimensions. Finally, we demonstrate that the effectiveness of algorithm portfolios depends on various factors. These findings provide fundamental insights for algorithm selection for black-box optimization.


page 1

page 2

page 3

page 4


Reviewing and Benchmarking Parameter Control Methods in Differential Evolution

Many Differential Evolution (DE) algorithms with various parameter contr...

Chaining of Numerical Black-box Algorithms: Warm-Starting and Switching Points

Dynamic algorithm selection can be beneficial for solving numerical blac...

A Feature Complete SPIKE Banded Algorithm and Solver

New features and enhancements for the SPIKE banded solver are presented....

ProvMark: A Provenance Expressiveness Benchmarking System

System level provenance is of widespread interest for applications such ...

COCOpf: An Algorithm Portfolio Framework

Algorithm portfolios represent a strategy of composing multiple heuristi...

COCO: Performance Assessment

We present an any-time performance assessment for benchmarking numerical...