From Local SGD to Local Fixed Point Methods for Federated Learning

by   Grigory Malinovsky, et al.

Most algorithms for solving optimization problems or finding saddle points of convex-concave functions are fixed point algorithms. In this work we consider the generic problem of finding a fixed point of an average of operators, or an approximation thereof, in a distributed setting. Our work is motivated by the needs of federated learning. In this context, each local operator models the computations done locally on a mobile device. We investigate two strategies to achieve such a consensus: one based on a fixed number of local steps, and the other based on randomized computations. In both cases, the goal is to limit communication of the locally-computed variables, which is often the bottleneck in distributed frameworks. We perform convergence analysis of both methods and conduct a number of experiments highlighting the benefits of our approach.



There are no comments yet.


page 1

page 2

page 3

page 4


Distributed Fixed Point Methods with Compressed Iterates

We propose basic and natural assumptions under which iterative optimizat...

FedSplit: An algorithmic framework for fast federated optimization

Motivated by federated learning, we consider the hub-and-spoke model of ...

ARock: an Algorithmic Framework for Asynchronous Parallel Coordinate Updates

Finding a fixed point to a nonexpansive operator, i.e., x^*=Tx^*, abstra...

On Second-order Optimization Methods for Federated Learning

We consider federated learning (FL), where the training data is distribu...

On Communication Complexity of Fixed Point Computation

Brouwer's fixed point theorem states that any continuous function from a...

An Operator Splitting View of Federated Learning

Over the past few years, the federated learning () community has witness...

Distributed Picard Iteration

The Picard iteration is widely used to find fixed points of locally cont...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.