On Convergence of FedProx: Local Dissimilarity Invariant Bounds, Non-smoothness and Beyond

06/10/2022
by   Xiao-Tong Yuan, et al.
0

The FedProx algorithm is a simple yet powerful distributed proximal point optimization method widely used for federated learning (FL) over heterogeneous data. Despite its popularity and remarkable success witnessed in practice, the theoretical understanding of FedProx is largely underinvestigated: the appealing convergence behavior of FedProx is so far characterized under certain non-standard and unrealistic dissimilarity assumptions of local functions, and the results are limited to smooth optimization problems. In order to remedy these deficiencies, we develop a novel local dissimilarity invariant convergence theory for FedProx and its minibatch stochastic extension through the lens of algorithmic stability. As a result, we contribute to derive several new and deeper insights into FedProx for non-convex federated optimization including: 1) convergence guarantees independent on local dissimilarity type conditions; 2) convergence guarantees for non-smooth FL problems; and 3) linear speedup with respect to size of minibatch and number of sampled devices. Our theory for the first time reveals that local dissimilarity and smoothness are not must-have for FedProx to get favorable complexity bounds. Preliminary experimental results on a series of benchmark FL datasets are reported to demonstrate the benefit of minibatching for improving the sample efficiency of FedProx.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/11/2020

Federated Learning's Blessing: FedAvg has Linear Speedup

Federated learning (FL) learns a model jointly from a set of participati...
research
01/27/2021

Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learning

Federated learning (FL) is a distributed machine learning architecture t...
research
01/27/2022

On the Convergence of Heterogeneous Federated Learning with Arbitrary Adaptive Online Model Pruning

One of the biggest challenges in Federated Learning (FL) is that client ...
research
12/25/2021

Towards Federated Learning on Time-Evolving Heterogeneous Data

Federated Learning (FL) is an emerging learning paradigm that preserves ...
research
09/25/2022

On the Stability Analysis of Open Federated Learning Systems

We consider the open federated learning (FL) systems, where clients may ...
research
05/23/2019

Scale Invariant Power Iteration

Power iteration has been generalized to solve many interesting problems ...
research
05/25/2023

Federated Composite Saddle Point Optimization

Federated learning (FL) approaches for saddle point problems (SPP) have ...

Please sign up or login with your details

Forgot password? Click here to reset