Log In Sign Up

Benchmarking Discrete Optimization Heuristics with IOHprofiler

by   Carola Doerr, et al.

Automated benchmarking environments aim to support researchers in understanding how different algorithms perform on different types of optimization problems. Such comparisons provide insights into the strengths and weaknesses of different approaches, which can be leveraged into designing new algorithms and into the automation of algorithm selection and configuration. With the ultimate goal to create a meaningful benchmark set for iterative optimization heuristics, we have recently released IOHprofiler, a software built to create detailed performance comparisons between iterative optimization heuristics. With this present work we demonstrate that IOHprofiler provides a suitable environment for automated benchmarking. We compile and assess a selection of 23 discrete optimization problems that subscribe to different types of fitness landscapes. For each selected problem we compare performances of twelve different heuristics, which are as of now available as baseline algorithms in IOHprofiler. We also provide a new module for IOHprofiler which extents the fixed-target and fixed-budget results for the individual problems by ECDF results, which allows one to derive aggregated performance statistics for groups of problems.


page 8

page 12

page 19

page 21

page 22

page 24

page 25

page 26


IOHexperimenter: Benchmarking Platform for Iterative Optimization Heuristics

We present IOHexperimenter, the experimentation module of the IOHprofile...

Extensible Logging and Empirical Attainment Function for IOHexperimenter

In order to allow for large-scale, landscape-aware, per-instance algorit...

Analyzing the Impact of Undersampling on the Benchmarking and Configuration of Evolutionary Algorithms

The stochastic nature of iterative optimization heuristics leads to inhe...

IOHprofiler: A Benchmarking and Profiling Tool for Iterative Optimization Heuristics

IOHprofiler is a new tool for analyzing and comparing iterative optimiza...

IOHanalyzer: Performance Analysis for Iterative Optimization Heuristic

We propose IOHanalyzer, a new software for analyzing the empirical perfo...

Towards Large Scale Automated Algorithm Design by Integrating Modular Benchmarking Frameworks

We present a first proof-of-concept use-case that demonstrates the effic...