Two-step hyperparameter optimization method: Accelerating hyperparameter search by using a fraction of a training dataset

02/08/2023
by   Sungduk Yu, et al.
0

Hyperparameter optimization (HPO) can be an important step in machine learning model development, but our common practice is archaic – primarily using a manual or grid search. This is partly because adopting an advanced HPO algorithm entails extra complexity to workflow and longer computation time. This imposes a significant hurdle to machine learning (ML) applications since the choice of suboptimal hyperparameters limits the performance of ML models, ultimately failing to harness the full potential of ML techniques. In this article, we present a two-step HPO method as a strategy to minimize compute and wait time as a lesson learned during applied ML parameterization work. A preliminary evaluation of hyperparameters is first conducted on a small subset of a training dataset, then top-performing candidate models are re-evaluated after retraining with an entire training dataset. This two-step HPO method can be applied to any HPO search algorithm, and we argue it has attractive efficiencies. As a case study, we present our recent application of the two-step HPO method to the development of neural network emulators of aerosol activation. Using only 5 sufficient to find optimal hyperparameter configurations from much more extensive sampling. The benefits of HPO are then revealed by analysis of hyperparameters and model performance, revealing a minimal model complexity required to achieve the best performance, and the diversity of top-performing models harvested from the HPO process allows us to choose a high-performing model with a low inference cost for efficient use in GCMs.

READ FULL TEXT

page 9

page 13

research
02/07/2015

Hyperparameter Search in Machine Learning

We introduce the hyperparameter search problem in the field of machine l...
research
11/12/2019

FLO: Fast and Lightweight Hyperparameter Optimization for AutoML

Integrating ML models in software is of growing interest. Building accur...
research
12/02/2020

VisEvol: Visual Analytics to Support Hyperparameter Search through Evolutionary Optimization

During the training phase of machine learning (ML) models, it is usually...
research
03/29/2018

An LP-based hyperparameter optimization model for language modeling

In order to find hyperparameters for a machine learning model, algorithm...
research
10/25/2020

Hyperparameter Transfer Across Developer Adjustments

After developer adjustments to a machine learning (ML) algorithm, how ca...
research
06/12/2019

Using Small Proxy Datasets to Accelerate Hyperparameter Search

One of the biggest bottlenecks in a machine learning workflow is waiting...
research
12/25/2019

A Study of the Learnability of Relational Properties (Model Counting Meets Machine Learning)

Relational properties, e.g., the connectivity structure of nodes in a di...

Please sign up or login with your details

Forgot password? Click here to reset