DeepAI AI Chat
Log In Sign Up

Enhanced Bilevel Optimization via Bregman Distance

by   Feihu Huang, et al.

Bilevel optimization has been widely applied many machine learning problems such as hyperparameter optimization, policy optimization and meta learning. Although many bilevel optimization methods more recently have been proposed to solve the bilevel optimization problems, they still suffer from high computational complexities and do not consider the more general bilevel problems with nonsmooth regularization. In the paper, thus, we propose a class of efficient bilevel optimization methods based on Bregman distance. In our methods, we use the mirror decent iteration to solve the outer subproblem of the bilevel problem by using strongly-convex Bregman functions. Specifically, we propose a bilevel optimization method based on Bregman distance (BiO-BreD) for solving deterministic bilevel problems, which reaches the lower computational complexities than the best known results. We also propose a stochastic bilevel optimization method (SBiO-BreD) for solving stochastic bilevel problems based on the stochastic approximated gradients and Bregman distance. Further, we propose an accelerated version of SBiO-BreD method (ASBiO-BreD) by using the variance-reduced technique. Moreover, we prove that the ASBiO-BreD outperforms the best known computational complexities with respect to the condition number κ and the target accuracy ϵ for finding an ϵ-stationary point of nonconvex-strongly-convex bilevel problems. In particular, our methods can solve the bilevel optimization problems with nonsmooth regularization with a lower computational complexity.


page 1

page 2

page 3

page 4


BiAdam: Fast Adaptive Bilevel Optimization Methods

Bilevel optimization recently has attracted increased interest in machin...

On Momentum-Based Gradient Methods for Bilevel Optimization with Nonconvex Lower-Level

Bilevel optimization is a popular two-level hierarchical optimization, w...

Provably Faster Algorithms for Bilevel Optimization and Applications to Meta-Learning

Bilevel optimization has arisen as a powerful tool for many machine lear...

Fast Adaptive Federated Bilevel Optimization

Bilevel optimization is a popular hierarchical model in machine learning...

Accelerated Zeroth-Order Momentum Methods from Mini to Minimax Optimization

In the paper, we propose a new accelerated zeroth-order momentum (Acc-ZO...

On Bilevel Optimization without Lower-level Strong Convexity

Theoretical properties of bilevel problems are well studied when the low...

Opytimizer: A Nature-Inspired Python Optimizer

Optimization aims at selecting a feasible set of parameters in an attemp...