Generalized Frank-Wolfe Algorithm for Bilevel Optimization

06/17/2022
by   Ruichen Jiang, et al.
0

In this paper, we study a class of bilevel optimization problems, also known as simple bilevel optimization, where we minimize a smooth objective function over the optimal solution set of another convex constrained optimization problem. Several iterative methods have been developed for tackling this class of problems. Alas, their convergence guarantees are not satisfactory as they are either asymptotic for the upper-level objective, or the convergence rates are slow and sub-optimal. To address this issue, in this paper, we introduce a generalization of the Frank-Wolfe (FW) method to solve the considered problem. The main idea of our method is to locally approximate the solution set of the lower-level problem via a cutting plane, and then run a FW-type update to decrease the upper-level objective. When the upper-level objective is convex, we show that our method requires 𝒪(max{1/ϵ_f,1/ϵ_g}) iterations to find a solution that is ϵ_f-optimal for the upper-level objective and ϵ_g-optimal for the lower-level objective. Moreover, when the upper-level objective is non-convex, our method requires 𝒪(max{1/ϵ_f^2,1/(ϵ_fϵ_g)}) iterations to find an (ϵ_f,ϵ_g)-optimal solution. We further prove stronger convergence guarantees under the Hölderian error bound assumption on the lower-level problem. To the best of our knowledge, our method achieves the best-known iteration complexity for the considered bilevel problem. We also present numerical experiments to showcase the superior performance of our method compared with state-of-the-art methods.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/15/2023

Projection-Free Methods for Stochastic Simple Bilevel Optimization with Convex Lower-level Problem

In this paper, we study a class of stochastic bilevel optimization probl...
research
02/15/2021

A Momentum-Assisted Single-Timescale Stochastic Approximation Algorithm for Bilevel Optimization

This paper proposes a new algorithm – the Momentum-assisted Single-times...
research
06/29/2020

kFW: A Frank-Wolfe style algorithm with stronger subproblem oracles

This paper proposes a new variant of Frank-Wolfe (FW), called kFW. Stand...
research
05/19/2021

Trilevel and Multilevel Optimization using Monotone Operator Theory

We consider rather a general class of multi-level optimization problems,...
research
01/25/2023

Optimisation of seismic imaging via bilevel learning

The implementation of Full Waveform Inversion (FWI) requires the a prior...
research
12/09/2015

RSG: Beating Subgradient Method without Smoothness and Strong Convexity

In this paper, we study the efficiency of a Restarted Sub Gradient (RS...
research
07/29/2023

Faster Stochastic Algorithms for Minimax Optimization under Polyak–Łojasiewicz Conditions

This paper considers stochastic first-order algorithms for minimax optim...

Please sign up or login with your details

Forgot password? Click here to reset