Understanding A Class of Decentralized and Federated Optimization Algorithms: A Multi-Rate Feedback Control Perspective

04/27/2022
by   Xinwei Zhang, et al.
0

Distributed algorithms have been playing an increasingly important role in many applications such as machine learning, signal processing, and control. Significant research efforts have been devoted to developing and analyzing new algorithms for various applications. In this work, we provide a fresh perspective to understand, analyze, and design distributed optimization algorithms. Through the lens of multi-rate feedback control, we show that a wide class of distributed algorithms, including popular decentralized/federated schemes, can be viewed as discretizing a certain continuous-time feedback control system, possibly with multiple sampling rates, such as decentralized gradient descent, gradient tracking, and federated averaging. This key observation not only allows us to develop a generic framework to analyze the convergence of the entire algorithm class. More importantly, it also leads to an interesting way of designing new distributed algorithms. We develop the theory behind our framework and provide examples to highlight how the framework can be used in practice.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/22/2021

FLoBC: A Decentralized Blockchain-Based Federated Learning Framework

The rapid expansion of data worldwide invites the need for more distribu...
research
06/20/2020

On the Divergence of Decentralized Non-Convex Optimization

We study a generic class of decentralized algorithms in which N agents j...
research
01/26/2022

A dual approach for federated learning

We study the federated optimization problem from a dual perspective and ...
research
06/06/2023

Stochastic Multi-Level Compositional Optimization Algorithms over Networks with Level-Independent Convergence Rate

Stochastic multi-level compositional optimization problems cover many ne...
research
09/12/2019

Communication-Efficient Distributed Optimization in Networks with Gradient Tracking

There is a growing interest in large-scale machine learning and optimiza...
research
06/14/2021

Decentralized Personalized Federated Min-Max Problems

Personalized Federated Learning has recently seen tremendous progress, a...
research
03/08/2023

ELF: Federated Langevin Algorithms with Primal, Dual and Bidirectional Compression

Federated sampling algorithms have recently gained great popularity in t...

Please sign up or login with your details

Forgot password? Click here to reset