Online Hyperparameter Search Interleaved with Proximal Parameter Updates

04/06/2020
by   Luis Miguel Lopez-Ramos, et al.
0

There is a clear need for efficient algorithms to tune hyperparameters for statistical learning schemes, since the commonly applied search methods (such as grid search with N-fold cross-validation) are inefficient and/or approximate. Previously existing algorithms that efficiently search for hyperparameters relying on the smoothness of the cost function cannot be applied in problems such as Lasso regression. In this contribution, we develop a hyperparameter optimization method that relies on the structure of proximal gradient methods and does not require a smooth cost function. Such a method is applied to Leave-one-out (LOO)-validated Lasso and Group Lasso to yield efficient, data-driven, hyperparameter optimization algorithms. Numerical experiments corroborate the convergence of the proposed method to a local optimum of the LOO validation error curve, and the efficiency of its approximations.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/21/2020

A Gradient-based Bilevel Optimization Approach for Tuning Hyperparameters in Machine Learning

Hyperparameter tuning is an active area of research in machine learning,...
research
02/20/2020

Implicit differentiation of Lasso-type models for hyperparameter optimization

Setting regularization parameters for Lasso-type estimators is notorious...
research
11/20/2020

Optimizing Approximate Leave-one-out Cross-validation to Tune Hyperparameters

For a large class of regularized models, leave-one-out cross-validation ...
research
07/13/2017

Discriminative Optimization: Theory and Applications to Computer Vision Problems

Many computer vision problems are formulated as the optimization of a co...
research
03/29/2018

An LP-based hyperparameter optimization model for language modeling

In order to find hyperparameters for a machine learning model, algorithm...
research
01/17/2021

Cost-Efficient Online Hyperparameter Optimization

Recent work on hyperparameters optimization (HPO) has shown the possibil...
research
10/30/2014

Robust sketching for multiple square-root LASSO problems

Many learning tasks, such as cross-validation, parameter search, or leav...

Please sign up or login with your details

Forgot password? Click here to reset