Improved Communication Lower Bounds for Distributed Optimisation

by   Dan Alistarh, et al.

Motivated by the interest in communication-efficient methods for distributed machine learning, we consider the communication complexity of minimising a sum of d-dimensional functions ∑_i = 1^N f_i (x), where each function f_i is held by a one of the N different machines. Such tasks arise naturally in large-scale optimisation, where a standard solution is to apply variants of (stochastic) gradient descent. As our main result, we show that Ω( Nd log d / ε) bits in total need to be communicated between the machines to find an additive ϵ-approximation to the minimum of ∑_i = 1^N f_i (x). The results holds for deterministic algorithms, and randomised algorithms under some restrictions on the parameter values. Importantly, our lower bounds require no assumptions on the structure of the algorithm, and are matched within constant factors for strongly convex objectives by a new variant of quantised gradient descent. The lower bounds are obtained by bringing over tools from communication complexity to distributed optimisation, an approach we hope will find further use in future.


page 1

page 2

page 3

page 4


The Minimax Complexity of Distributed Optimization

In this thesis, I study the minimax oracle complexity of distributed sto...

AIDE: Fast and Communication Efficient Distributed Optimization

In this paper, we present two new communication-efficient methods for di...

Distributed Estimation of Generalized Matrix Rank: Efficient Algorithms and Lower Bounds

We study the following generalized matrix rank estimation problem: given...

Theoretical Limits of One-Shot Distributed Learning

We consider a distributed system of m machines and a server. Each machin...

Tight Complexity Bounds for Optimizing Composite Objectives

We provide tight upper and lower bounds on the complexity of minimizing ...

Convex Set Disjointness, Distributed Learning of Halfspaces, and LP Feasibility

We study the Convex Set Disjointness (CSD) problem, where two players ha...

On Fundamental Limits of Robust Learning

We consider the problems of robust PAC learning from distributed and str...