Order Optimal One-Shot Distributed Learning

11/02/2019
by   Arsalan Sharifnassab, et al.
0

We consider distributed statistical optimization in one-shot setting, where there are m machines each observing n i.i.d. samples. Based on its observed samples, each machine then sends an O(log(mn))-length message to a server, at which a parameter minimizing an expected loss is to be estimated. We propose an algorithm called Multi-Resolution Estimator (MRE) whose expected error is no larger than Õ(m^-1/max(d,2) n^-1/2), where d is the dimension of the parameter space. This error bound meets existing lower bounds up to poly-logarithmic factors, and is thereby order optimal. The expected error of MRE, unlike existing algorithms, tends to zero as the number of machines (m) goes to infinity, even when the number of samples per machine (n) remains upper bounded by a constant. This property of the MRE algorithm makes it applicable in new machine learning paradigms where m is much larger than n.

READ FULL TEXT

Authors

page 1

page 2

page 3

page 4

05/12/2019

Theoretical Limits of One-Shot Distributed Learning

We consider a distributed system of m machines and a server. Each machin...
08/19/2021

Order Optimal One-Shot Federated Learning for non-Convex Loss Functions

We consider the problem of federated learning in a one-shot setting in w...
01/24/2022

Optimal SQ Lower Bounds for Learning Halfspaces with Massart Noise

We give tight statistical query (SQ) lower bounds for learnining halfspa...
02/28/2019

Distributed Learning with Sublinear Communication

In distributed statistical learning, N samples are split across m machin...
01/10/2019

Mean Estimation from One-Bit Measurements

We consider the problem of estimating the mean of a symmetric log-concav...
11/09/2021

Towards a Unified Information-Theoretic Framework for Generalization

In this work, we investigate the expressiveness of the "conditional mutu...
04/05/2019

Parameter estimation for integer-valued Gibbs distributions

We consider the family of Gibbs distributions, which are probability dis...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.