From Local SGD to Local Fixed Point Methods for Federated Learning

04/03/2020
by   Grigory Malinovsky, et al.
0

Most algorithms for solving optimization problems or finding saddle points of convex-concave functions are fixed point algorithms. In this work we consider the generic problem of finding a fixed point of an average of operators, or an approximation thereof, in a distributed setting. Our work is motivated by the needs of federated learning. In this context, each local operator models the computations done locally on a mobile device. We investigate two strategies to achieve such a consensus: one based on a fixed number of local steps, and the other based on randomized computations. In both cases, the goal is to limit communication of the locally-computed variables, which is often the bottleneck in distributed frameworks. We perform convergence analysis of both methods and conduct a number of experiments highlighting the benefits of our approach.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

12/20/2019

Distributed Fixed Point Methods with Compressed Iterates

We propose basic and natural assumptions under which iterative optimizat...
05/11/2020

FedSplit: An algorithmic framework for fast federated optimization

Motivated by federated learning, we consider the hub-and-spoke model of ...
06/08/2015

ARock: an Algorithmic Framework for Asynchronous Parallel Coordinate Updates

Finding a fixed point to a nonexpansive operator, i.e., x^*=Tx^*, abstra...
09/06/2021

On Second-order Optimization Methods for Federated Learning

We consider federated learning (FL), where the training data is distribu...
09/24/2019

On Communication Complexity of Fixed Point Computation

Brouwer's fixed point theorem states that any continuous function from a...
08/12/2021

An Operator Splitting View of Federated Learning

Over the past few years, the federated learning () community has witness...
03/31/2021

Distributed Picard Iteration

The Picard iteration is widely used to find fixed points of locally cont...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.