DeepAI AI Chat
Log In Sign Up

Black-Box Optimization Revisited: Improving Algorithm Selection Wizards through Massive Benchmarking

by   Laurent Meunier, et al.

Existing studies in black-box optimization suffer from low generalizability, caused by a typically selective choice of problem instances used for training and testing different optimization algorithms. Among other issues, this practice promotes overfitting and poor-performing user guide-lines. To address this shortcoming, we propose in this work a benchmark suite which covers a broad range of black-box optimization problems, ranging from academic benchmarks to real-world optimization problems, from discrete over numerical to mixed-integer problems, from small to very large-scale problems, from noisy over dynamic to static problems, etc. We demonstrate the advantages of such a broad collection by deriving from it NGOpt8, a general-purpose algorithm selection wizard. Using three different types of algorithm selection techniques, NGOpt8 achieves competitive performance on all benchmark suites. It significantly outperforms previous state of the art on some of them, including the MuJoCo collection,YABBOB, and LSGO. A single algorithm therefore performed best on these three important benchmarks, without any task-specific parametrization. The benchmark collection, the wizard, its low-level solvers, as well as all experimental data are fully reproducible and open source. They are made available as a fork of Nevergrad, termed OptimSuite.


page 1

page 2

page 3

page 4


Benchmarking for Metaheuristic Black-Box Optimization: Perspectives and Open Challenges

Research on new optimization algorithms is often funded based on the mot...

Black Box Algorithm Selection by Convolutional Neural Network

Although a large number of optimization algorithms have been proposed fo...

Cardinality Minimization, Constraints, and Regularization: A Survey

We survey optimization problems that involve the cardinality of variable...

Generating Large-scale Dynamic Optimization Problem Instances Using the Generalized Moving Peaks Benchmark

This document describes the generalized moving peaks benchmark (GMPB) an...

Engineering a Preprocessor for Symmetry Detection

State-of-the-art solvers for symmetry detection in combinatorial objects...

DQI: A Guide to Benchmark Evaluation

A `state of the art' model A surpasses humans in a benchmark B, but fail...