DeepAI AI Chat
Log In Sign Up

Asynchronous Distributed Bilevel Optimization

by   Yang Jiao, et al.

Bilevel optimization plays an essential role in many machine learning tasks, ranging from hyperparameter optimization to meta-learning. Existing studies on bilevel optimization, however, focus on either centralized or synchronous distributed setting. The centralized bilevel optimization approaches require collecting massive amount of data to a single server, which inevitably incur significant communication expenses and may give rise to data privacy risks. Synchronous distributed bilevel optimization algorithms, on the other hand, often face the straggler problem and will immediately stop working if a few workers fail to respond. As a remedy, we propose Asynchronous Distributed Bilevel Optimization (ADBO) algorithm. The proposed ADBO can tackle bilevel optimization problems with both nonconvex upper-level and lower-level objective functions, and its convergence is theoretically guaranteed. Furthermore, it is revealed through theoretic analysis that the iteration complexity of ADBO to obtain the ϵ-stationary point is upper bounded by 𝒪(1/ϵ ^2). Thorough empirical studies on public datasets have been conducted to elucidate the effectiveness and efficiency of the proposed ADBO.


page 1

page 2

page 3

page 4


Fast Adaptive Federated Bilevel Optimization

Bilevel optimization is a popular hierarchical model in machine learning...

Convergence Properties of Stochastic Hypergradients

Bilevel optimization problems are receiving increasing attention in mach...

Impact of Communication Delay on Asynchronous Distributed Optimal Power Flow Using ADMM

Distributed optimization has attracted lots of attention in the operatio...

Distributed Distributionally Robust Optimization with Non-Convex Objectives

Distributionally Robust Optimization (DRO), which aims to find an optima...

Asynchronous Distributed Optimization with Randomized Delays

In this work, we study asynchronous finite sum minimization in a distrib...

Federated Distributionally Robust Optimization with Non-Convex Objectives: Algorithm and Analysis

Distributionally Robust Optimization (DRO), which aims to find an optima...

Deep learning with Elastic Averaging SGD

We study the problem of stochastic optimization for deep learning in the...