Bilevel Optimization with a Lower-level Contraction: Optimal Sample Complexity without Warm-Start

02/07/2022
by   Riccardo Grazzi, et al.
0

We analyze a general class of bilevel problems, in which the upper-level problem consists in the minimization of a smooth objective function and the lower-level problem is to find the fixed point of a smooth contraction map. This type of problems include instances of meta-learning, hyperparameter optimization and data poisoning adversarial attacks. Several recent works have proposed algorithms which warm-start the lower-level problem, i.e. they use the previous lower-level approximate solution as a staring point for the lower-level solver. This warm-start procedure allows one to improve the sample complexity in both the stochastic and deterministic settings, achieving in some cases the order-wise optimal sample complexity. We show that without warm-start, it is still possible to achieve order-wise optimal and near-optimal sample complexity for the stochastic and deterministic settings, respectively. In particular, we propose a simple method which uses stochastic fixed point iterations at the lower-level and projected inexact gradient descent at the upper-level, that reaches an ϵ-stationary point using O(ϵ^-2) and Õ(ϵ^-1) samples for the stochastic and the deterministic setting, respectively. Compared to methods using warm-start, ours is better suited for meta-learning and yields a simpler analysis that does not need to study the coupled interactions between the upper-level and lower-level iterates.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/09/2021

A Single-Timescale Stochastic Bilevel Optimization Method

Stochastic bilevel optimization generalizes the classic stochastic optim...
research
06/29/2020

On the Iteration Complexity of Hypergradient Computation

We study a general class of bilevel problems, consisting in the minimiza...
research
11/13/2020

Convergence Properties of Stochastic Hypergradients

Bilevel optimization problems are receiving increasing attention in mach...
research
02/17/2023

A Near-Optimal Algorithm for Bilevel Empirical Risk Minimization

Bilevel optimization problems, which are problems where two optimization...
research
06/26/2023

Near-Optimal Fully First-Order Algorithms for Finding Stationary Points in Bilevel Optimization

Bilevel optimization has various applications such as hyper-parameter op...
research
10/31/2022

Private optimization in the interpolation regime: faster rates and hardness results

In non-private stochastic convex optimization, stochastic gradient metho...
research
02/14/2022

Stochastic linear optimization never overfits with quadratically-bounded losses on general data

This work shows that a diverse collection of linear optimization methods...

Please sign up or login with your details

Forgot password? Click here to reset