Towards Feature-Based Performance Regression Using Trajectory Data

02/10/2021
by   Anja Jankovic, et al.
0

Black-box optimization is a very active area of research, with many new algorithms being developed every year. This variety is needed, on the one hand, since different algorithms are most suitable for different types of optimization problems. But the variety also poses a meta-problem: which algorithm to choose for a given problem at hand? Past research has shown that per-instance algorithm selection based on exploratory landscape analysis (ELA) can be an efficient mean to tackle this meta-problem. Existing approaches, however, require the approximation of problem features based on a significant number of samples, which are typically selected through uniform sampling or Latin Hypercube Designs. The evaluation of these points is costly, and the benefit of an ELA-based algorithm selection over a default algorithm must therefore be significant in order to pay off. One could hope to by-pass the evaluations for the feature approximations by using the samples that a default algorithm would anyway perform, i.e., by using the points of the default algorithm's trajectory. We analyze in this paper how well such an approach can work. Concretely, we test how accurately trajectory-based ELA approaches can predict the final solution quality of the CMA-ES after a fixed budget of function evaluations. We observe that the loss of trajectory-based predictions can be surprisingly small compared to the classical global sampling approach, if the remaining budget for which solution quality shall be predicted is not too large. Feature selection, in contrast, did not show any advantage in our experiments and rather led to worsened prediction accuracy. The inclusion of state variables of CMA-ES only has a moderate effect on the prediction accuracy.

READ FULL TEXT

page 6

page 7

page 9

04/13/2022

Trajectory-based Algorithm Selection with Warm-starting

Landscape-aware algorithm selection approaches have so far mostly been r...
06/17/2020

Landscape-Aware Fixed-Budget Performance Regression and Algorithm Selection for Modular CMA-ES Variants

Automated algorithm selection promises to support the user in the decisi...
04/20/2022

Per-run Algorithm Selection with Warm-starting using Trajectory-based Features

Per-instance algorithm selection seeks to recommend, for a given problem...
12/22/2019

Black Box Algorithm Selection by Convolutional Neural Network

Although a large number of optimization algorithms have been proposed fo...
06/19/2020

Exploratory Landscape Analysis is Strongly Sensitive to the Sampling Strategy

Exploratory landscape analysis (ELA) supports supervised learning approa...
04/22/2021

Personalizing Performance Regression Models to Black-Box Optimization Problems

Accurately predicting the performance of different optimization algorith...
02/01/2021

Towards Explainable Exploratory Landscape Analysis: Extreme Feature Selection for Classifying BBOB Functions

Facilitated by the recent advances of Machine Learning (ML), the automat...