MURANA: A Generic Framework for Stochastic Variance-Reduced Optimization

06/06/2021
by   Laurent Condat, et al.
12

We propose a generic variance-reduced algorithm, which we call MUltiple RANdomized Algorithm (MURANA), for minimizing a sum of several smooth functions plus a regularizer, in a sequential or distributed manner. Our method is formulated with general stochastic operators, which allow us to model various strategies for reducing the computational complexity. For example, MURANA supports sparse activation of the gradients, and also reduction of the communication load via compression of the update vectors. This versatility allows MURANA to cover many existing randomization mechanisms within a unified framework. However, MURANA also encodes new methods as special cases. We highlight one of them, which we call ELVIRA, and show that it improves upon Loopless SVRG.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/15/2022

Stochastic Gradient Descent-Ascent: Unified Theory and New Efficient Methods

Stochastic Gradient Descent-Ascent (SGDA) is one of the most prominent a...
research
06/20/2020

Unified Analysis of Stochastic Gradient Methods for Composite Convex and Smooth Optimization

We present a unified theorem for the convergence analysis of stochastic ...
research
07/21/2019

Distributed Inexact Successive Convex Approximation ADMM: Analysis-Part I

In this two-part work, we propose an algorithmic framework for solving n...
research
02/13/2020

Sampling and Update Frequencies in Proximal Variance Reduced Stochastic Gradient Methods

Variance reduced stochastic gradient methods have gained popularity in r...
research
08/04/2021

ErrorCompensatedX: error compensation for variance reduced algorithms

Communication cost is one major bottleneck for the scalability for distr...
research
05/27/2019

One Method to Rule Them All: Variance Reduction for Data, Parameters and Many New Methods

We propose a remarkably general variance-reduced method suitable for sol...
research
06/21/2022

Shifted Compression Framework: Generalizations and Improvements

Communication is one of the key bottlenecks in the distributed training ...

Please sign up or login with your details

Forgot password? Click here to reset