Order Optimal One-Shot Federated Learning for non-Convex Loss Functions

by   Arsalan Sharifnassab, et al.

We consider the problem of federated learning in a one-shot setting in which there are m machines, each observing n samples function from an unknown distribution on non-convex loss functions. Let F:[-1,1]^d→ℝ be the expected loss function with respect to this unknown distribution. The goal is to find an estimate of the minimizer of F. Based on its observations, each machine generates a signal of bounded length B and sends it to a server. The sever collects signals of all machines and outputs an estimate of the minimizer of F. We propose a distributed learning algorithm, called Multi-Resolution Estimator for Non-Convex loss function (MRE-NC), whose expected error is bounded by max(1/√(n)(mB)^1/d, 1/√(mn)), up to polylogarithmic factors. We also provide a matching lower bound on the performance of any algorithm, showing that MRE-NC is order optimal in terms of n and m. Experiments on synthetic and real data show the effectiveness of MRE-NC in distributed learning of model's parameters for non-convex loss functions.


page 1

page 2

page 3

page 4


Order Optimal One-Shot Distributed Learning

We consider distributed statistical optimization in one-shot setting, wh...

Personalized Federated Learning: A Meta-Learning Approach

The goal of federated learning is to design algorithms in which several ...

Theoretical Limits of One-Shot Distributed Learning

We consider a distributed system of m machines and a server. Each machin...

Private Non-Convex Federated Learning Without a Trusted Server

We study differentially private (DP) federated learning (FL) with non-co...

Iterated Vector Fields and Conservatism, with Applications to Federated Learning

We study iterated vector fields and investigate whether they are conserv...

Persistent Reductions in Regularized Loss Minimization for Variable Selection

In the context of regularized loss minimization with polyhedral gauges, ...

Online Non-convex Learning for River Pollution Source Identification

In this paper, novel gradient based online learning algorithms are devel...