Order Optimal One-Shot Federated Learning for non-Convex Loss Functions

08/19/2021
by   Arsalan Sharifnassab, et al.
0

We consider the problem of federated learning in a one-shot setting in which there are m machines, each observing n samples function from an unknown distribution on non-convex loss functions. Let F:[-1,1]^d→ℝ be the expected loss function with respect to this unknown distribution. The goal is to find an estimate of the minimizer of F. Based on its observations, each machine generates a signal of bounded length B and sends it to a server. The sever collects signals of all machines and outputs an estimate of the minimizer of F. We propose a distributed learning algorithm, called Multi-Resolution Estimator for Non-Convex loss function (MRE-NC), whose expected error is bounded by max(1/√(n)(mB)^1/d, 1/√(mn)), up to polylogarithmic factors. We also provide a matching lower bound on the performance of any algorithm, showing that MRE-NC is order optimal in terms of n and m. Experiments on synthetic and real data show the effectiveness of MRE-NC in distributed learning of model's parameters for non-convex loss functions.

READ FULL TEXT

page 1

page 2

page 3

page 4

11/02/2019

Order Optimal One-Shot Distributed Learning

We consider distributed statistical optimization in one-shot setting, wh...
02/19/2020

Personalized Federated Learning: A Meta-Learning Approach

The goal of federated learning is to design algorithms in which several ...
05/12/2019

Theoretical Limits of One-Shot Distributed Learning

We consider a distributed system of m machines and a server. Each machin...
03/13/2022

Private Non-Convex Federated Learning Without a Trusted Server

We study differentially private (DP) federated learning (FL) with non-co...
09/08/2021

Iterated Vector Fields and Conservatism, with Applications to Federated Learning

We study iterated vector fields and investigate whether they are conserv...
11/30/2020

Persistent Reductions in Regularized Loss Minimization for Variable Selection

In the context of regularized loss minimization with polyhedral gauges, ...
05/22/2020

Online Non-convex Learning for River Pollution Source Identification

In this paper, novel gradient based online learning algorithms are devel...