Benchmarking in Optimization: Best Practice and Open Issues

by   Thomas Bartz-Beielstein, et al.

This survey compiles ideas and recommendations from more than a dozen researchers with different backgrounds and from different institutes around the world. Promoting best practice in benchmarking is its main goal. The article discusses eight essential topics in benchmarking: clearly stated goals, well-specified problems, suitable algorithms, adequate performance measures, thoughtful analysis, effective and efficient designs, comprehensible presentations, and guaranteed reproducibility. The final goal is to provide well-accepted guidelines (rules) that might be useful for authors and reviewers. As benchmarking in optimization is an active and evolving field of research this manuscript is meant to co-evolve over time by means of periodic updates.


page 6

page 7

page 8


Essential guidelines for computational method benchmarking

In computational biology and other sciences, researchers are frequently ...

Benchmarking Crimes: An Emerging Threat in Systems Security

Properly benchmarking a system is a difficult and intricate task. Unfort...

Building benchmarking frameworks for supporting replicability and reproducibility: spatial and textual analysis as an example

Replicability and reproducibility (R R) are critical for the long-term...

Guidelines for benchmarking of optimization approaches for fitting mathematical models

Insufficient performance of optimization approaches for fitting of mathe...

A survey of benchmarking frameworks for reinforcement learning

Reinforcement learning has recently experienced increased prominence in ...

Dockerization Impacts in Database Performance Benchmarking

Docker seems to be an attractive solution for cloud database benchmarkin...

IOHexperimenter: Benchmarking Platform for Iterative Optimization Heuristics

We present IOHexperimenter, the experimentation module of the IOHprofile...