FEDNEST: Federated Bilevel, Minimax, and Compositional Optimization

05/04/2022
by   Davoud Ataee Tarzanagh, et al.
0

Standard federated optimization methods successfully apply to stochastic problems with single-level structure. However, many contemporary ML problems – including adversarial robustness, hyperparameter tuning, and actor-critic – fall under nested bilevel programming that subsumes minimax and compositional optimization. In this work, we propose FedNest: A federated alternating stochastic gradient method to address general nested problems. We establish provable convergence rates for FedNest in the presence of heterogeneous data and introduce variations for bilevel, minimax, and compositional optimization. FedNest introduces multiple innovations including federated hypergradient computation and variance reduction to address inner-level heterogeneity. We complement our theory with experiments on hyperparameter & hyper-representation learning and minimax optimization that demonstrate the benefits of our method in practice. Code is available at https://github.com/mc-nya/FedNest.

READ FULL TEXT
research
06/02/2023

Federated Multi-Sequence Stochastic Approximation with Local Hypergradient Estimation

Stochastic approximation with multiple coupled sequences (MSA) has found...
research
11/14/2022

Adaptive Federated Minimax Optimization with Lower complexities

Federated learning is a popular distributed and privacy-preserving machi...
research
08/18/2023

Breaking the Complexity Barrier in Compositional Minimax Optimization

Compositional minimax optimization is a pivotal yet under-explored chall...
research
06/25/2021

Tighter Analysis of Alternating Stochastic Gradient Method for Stochastic Nested Problems

Stochastic nested optimization, including stochastic compositional, min-...
research
06/21/2021

Compositional Federated Learning: Applications in Distributionally Robust Averaging and Meta Learning

In the paper, we propose an effective and efficient Compositional Federa...
research
07/25/2023

Achieving Linear Speedup in Decentralized Stochastic Compositional Minimax Optimization

The stochastic compositional minimax problem has attracted a surge of at...
research
04/20/2023

Federated Compositional Deep AUC Maximization

Federated learning has attracted increasing attention due to the promise...

Please sign up or login with your details

Forgot password? Click here to reset