Dynamic Bilevel Learning with Inexact Line Search

by   Mohammad Sadegh Salehi, et al.

In various domains within imaging and data science, particularly when addressing tasks modeled utilizing the variational regularization approach, manually configuring regularization parameters presents a formidable challenge. The difficulty intensifies when employing regularizers involving a large number of hyperparameters. To overcome this challenge, bilevel learning is employed to learn suitable hyperparameters. However, due to the use of numerical solvers, the exact gradient with respect to the hyperparameters is unattainable, necessitating the use of methods relying on approximate gradients. State-of-the-art inexact methods a priori select a decreasing summable sequence of the required accuracy and only assure convergence given a sufficiently small fixed step size. Despite this, challenges persist in determining the Lipschitz constant of the hypergradient and identifying an appropriate fixed step size. Conversely, computing exact function values is not feasible, impeding the use of line search. In this work, we introduce a provably convergent inexact backtracking line search involving inexact function evaluations and hypergradients. We show convergence to a stationary point of the loss with respect to hyperparameters. Additionally, we propose an algorithm to determine the required accuracy dynamically. Our numerical experiments demonstrate the efficiency and feasibility of our approach for hyperparameter estimation in variational regularization problems, alongside its robustness in terms of the initial accuracy and step size choices.


page 20

page 22


Gradient-based Hyperparameter Optimization through Reversible Learning

Tuning hyperparameters of learning algorithms is hard because gradients ...

A Data-Driven Line Search Rule for Support Recovery in High-dimensional Data Analysis

In this work, we consider the algorithm to the (nonlinear) regression pr...

A Constant Step Stochastic Douglas-Rachford Algorithm with Application to Non Separable Regularizations

The Douglas Rachford algorithm is an algorithm that converges to a minim...

Fast Line Search for Multi-Task Learning

Multi-task learning is a powerful method for solving several tasks joint...

Hyperparameter optimization with approximate gradient

Most models in machine learning contain at least one hyperparameter to c...

Regularized Barzilai-Borwein method

This paper is concerned with the introduction of regularization into RBB...

Margins, Shrinkage, and Boosting

This manuscript shows that AdaBoost and its immediate variants can produ...

Please sign up or login with your details

Forgot password? Click here to reset