DeepAI AI Chat
Log In Sign Up

Hyperparameter optimization with approximate gradient

02/07/2016
by   Fabian Pedregosa, et al.
0

Most models in machine learning contain at least one hyperparameter to control for model complexity. Choosing an appropriate set of hyperparameters is both crucial in terms of model accuracy and computationally challenging. In this work we propose an algorithm for the optimization of continuous hyperparameters using inexact gradient information. An advantage of this method is that hyperparameters can be updated before model parameters have fully converged. We also give sufficient conditions for the global convergence of this method, based on regularity conditions of the involved functions and summability of errors. Finally, we validate the empirical performance of this method on the estimation of regularization constants of L2-regularized logistic regression and kernel Ridge regression. Empirical benchmarks indicate that our approach is highly competitive with respect to state of the art methods.

READ FULL TEXT

page 1

page 2

page 3

page 4

02/14/2018

Stealing Hyperparameters in Machine Learning

Hyperparameters are critical in machine learning, as different hyperpara...
05/13/2019

Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization

Due to the high computational demands executing a rigorous comparison be...
08/25/2022

A Globally Convergent Gradient-based Bilevel Hyperparameter Optimization Method

Hyperparameter optimization in machine learning is often achieved using ...
01/17/2022

Efficient Hyperparameter Tuning for Large Scale Kernel Ridge Regression

Kernel methods provide a principled approach to nonparametric learning. ...
05/23/2021

Regularization Can Help Mitigate Poisoning Attacks... with the Right Hyperparameters

Machine learning algorithms are vulnerable to poisoning attacks, where a...
05/05/2023

Optimizing Hyperparameters with Conformal Quantile Regression

Many state-of-the-art hyperparameter optimization (HPO) algorithms rely ...

Code Repositories

hoag

Hyperparameter optimization with approximate gradient


view repo