HPO: We won't get fooled again

08/04/2022
by   Kalifou René Traoré, et al.
0

Hyperparameter optimization (HPO) is a well-studied research field. However, the effects and interactions of the components in an HPO pipeline are not yet well investigated. Then, we ask ourselves: can the landscape of HPO be biased by the pipeline used to evaluate individual configurations? To address this question, we proposed to analyze the effect of the HPO pipeline on HPO problems using fitness landscape analysis. Particularly, we studied the DS-2019 HPO benchmark data set, looking for patterns that could indicate evaluation pipeline malfunction, and relate them to HPO performance. Our main findings are: (i) In most instances, large groups of diverse hyperparameters (i.e., multiple configurations) yield the same ill performance, most likely associated with majority class prediction models; (ii) in these cases, a worsened correlation between the observed fitness and average fitness in the neighborhood is observed, potentially making harder the deployment of local-search based HPO strategies. Finally, we concluded that the HPO pipeline definition might negatively affect the HPO landscape.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/18/2012

Set-based Multiobjective Fitness Landscapes: A Preliminary Study

Fitness landscape analysis aims to understand the geometry of a given op...
research
04/29/2020

Multi-layer local optima networks for the analysis of advanced local search-based algorithms

A Local Optima Network (LON) is a graph model that compresses the fitnes...
research
12/21/2016

Scale-invariance of ruggedness measures in fractal fitness landscapes

The paper deals with using chaos to direct trajectories to targets and a...
research
04/30/2018

New Methods of Studying Valley Fitness Landscapes

The word "valley" is a popular term used in intuitively describing fitne...
research
04/26/2021

An Algorithm to Effect Prompt Termination of Myopic Local Search on Kauffman-s NK Landscape

In the NK model given by Kauffman, myopic local search involves flipping...
research
05/10/2021

Overcoming Complexity Catastrophe: An Algorithm for Beneficial Far-Reaching Adaptation under High Complexity

In his seminal work with NK algorithms, Kauffman noted that fitness outc...
research
06/16/2023

On Evolvability and Behavior Landscapes in Neuroevolutionary Divergent Search

Evolvability refers to the ability of an individual genotype (solution) ...

Please sign up or login with your details

Forgot password? Click here to reset