research
∙
06/28/2023
Towards a Better Theoretical Understanding of Independent Subnetwork Training
Modern advancements in large-scale machine learning would be impossible ...
share
research
∙
06/06/2023
Understanding Progressive Training Through the Framework of Randomized Coordinate Descent
We propose a Randomized Progressive Training algorithm (RPT) – a stochas...
research
∙
06/05/2023
Improving Accelerated Federated Learning with Compression and Importance Sampling
Federated Learning is a collaborative training framework that leverages ...
research
∙
05/30/2023
Clip21: Error Feedback for Gradient Clipping
Motivated by the increasing popularity and importance of large-scale tra...
research
∙
05/29/2023
Global-QSGD: Practical Floatless Quantization for Distributed Learning with Theoretical Guarantees
Efficient distributed training is a principal driver of recent advances ...
research
∙
05/25/2023
A Guide Through the Zoo of Biased SGD
Stochastic Gradient Descent (SGD) is arguably the most important single ...
research
∙
05/24/2023
Momentum Provably Improves Error Feedback!
Due to the high communication overhead when training machine learning mo...
research
∙
03/08/2023
ELF: Federated Langevin Algorithms with Primal, Dual and Bidirectional Compression
Federated sampling algorithms have recently gained great popularity in t...
research
∙
02/20/2023
TAMUNA: Accelerated Federated Learning with Local Training and Partial Participation
In federated learning, a large number of users are involved in a global ...
research
∙
01/17/2023
Convergence of First-Order Algorithms for Meta-Learning with Moreau Envelopes
In this work, we consider the problem of minimizing the sum of Moreau en...
research
∙
12/29/2022
Can 5th Generation Local Training Methods Support Client Sampling? Yes!
The celebrated FedAvg algorithm of McMahan et al. (2017) is based on thr...
research
∙
10/31/2022
Adaptive Compression for Communication-Efficient Distributed Training
We propose Adaptive Compressed Gradient Descent (AdaCGD) - a novel optim...
research
∙
10/28/2022
GradSkip: Communication-Accelerated Local Gradient Methods with Better Computational Complexity
In this work, we study distributed optimization algorithms that reduce t...
research
∙
10/24/2022
Provably Doubly Accelerated Federated Learning: The First Theoretically Successful Combination of Local Training and Compressed Communication
In the modern paradigm of federated learning, a large number of users ar...
research
∙
10/02/2022
Improved Stein Variational Gradient Descent with Importance Weights
Stein Variational Gradient Descent (SVGD) is a popular sampling algorith...
research
∙
09/30/2022
EF21-P and Friends: Improved Theoretical Communication Complexity for Distributed Optimization with Bidirectional Compression
The starting point of this paper is the discovery of a novel and simple ...
research
∙
09/16/2022
Minibatch Stochastic Three Points Method for Unconstrained Smooth Minimization
In this paper, we propose a new zero order optimization method called mi...
research
∙
08/10/2022
Adaptive Learning Rates for Faster Stochastic Gradient Methods
In this work, we propose new adaptive step size strategies that improve ...
research
∙
07/08/2022
Communication Acceleration of Local Gradient Methods via an Accelerated Primal-Dual Algorithm with Inexact Prox
Inspired by a recent breakthrough of Mishchenko et al (2022), who for th...
research
∙
06/21/2022
Shifted Compression Framework: Generalizations and Improvements
Communication is one of the key bottlenecks in the distributed training ...
research
∙
06/20/2022
A Note on the Convergence of Mirrored Stein Variational Gradient Descent under (L_0,L_1)-Smoothness Condition
In this note, we establish a descent lemma for the population limit Mirr...
research
∙
06/07/2022
Distributed Newton-Type Methods with Communication Compression and Bernoulli Aggregation
Despite their high computation and communication costs, Newton-type meth...
research
∙
06/06/2022
Certified Robustness in Federated Learning
Federated learning has recently gained significant attention and popular...
research
∙
06/02/2022
Federated Learning with a Sampling Algorithm under Isoperimetry
Federated learning uses a set of techniques to efficiently distribute th...
research
∙
06/01/2022
Variance Reduction is an Antidote to Byzantines: Better Rates, Weaker Assumptions and Communication Compression as a Cherry on the Top
Byzantine-robustness has been gaining a lot of attention due to the grow...
research
∙
06/01/2022
Convergence of Stein Variational Gradient Descent under a Weaker Smoothness Condition
Stein Variational Gradient Descent (SVGD) is an important alternative to...
research
∙
05/31/2022
A Computation and Communication Efficient Method for Distributed Nonconvex Problems in the Partial Participation Setting
We present a new method that includes three key components of distribute...
research
∙
05/08/2022
Federated Random Reshuffling with Compression and Variance Reduction
Random Reshuffling (RR), which is a variant of Stochastic Gradient Desce...
research
∙
04/27/2022
FedShuffle: Recipes for Better Use of Local Work in Federated Learning
The practice of applying several local updates before aggregation across...
research
∙
02/18/2022
ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!
We introduce ProxSkip – a surprisingly simple and provably efficient met...
research
∙
02/07/2022
FL_PyTorch: optimization research simulator for federated learning
Federated Learning (FL) has emerged as a promising technique for edge de...
research
∙
02/02/2022
DASHA: Distributed Nonconvex Optimization with Communication Compression, Optimal Oracle Complexity, and No Client Synchronization
We develop and analyze DASHA: a new family of methods for nonconvex dist...
research
∙
02/02/2022
3PC: Three Point Compressors for Communication-Efficient Distributed Training and a Better Theory for Lazy Aggregation
We propose and study a new class of gradient communication mechanisms fo...
research
∙
01/31/2022
BEER: Fast O(1/T) Rate for Decentralized Nonconvex Optimization with Communication Compression
Communication efficiency has been widely recognized as the bottleneck fo...
research
∙
01/26/2022
Server-Side Stepsizes and Sampling Without Replacement Provably Help in Federated Optimization
We present a theoretical study of server-side optimization in federated ...
research
∙
12/24/2021
Faster Rates for Compressed Federated Learning with Client-Variance Reduction
Due to the communication bottleneck in distributed and federated learnin...
research
∙
11/22/2021
FLIX: A Simple and Communication-Efficient Alternative to Local Methods in Federated Learning
Federated Learning (FL) is an increasingly popular machine learning para...
research
∙
11/02/2021
Basis Matters: Better Communication-Efficient Second Order Methods for Federated Learning
Recent advances in distributed optimization have shown that Newton-type ...
research
∙
10/07/2021
Permutation Compressors for Provably Faster Distributed Nonconvex Optimization
We study the MARINA method of Gorbunov et al (2021) – the current state-...
research
∙
10/07/2021
EF21 with Bells Whistles: Practical Algorithmic Extensions of Modern Error Feedback
First proposed by Seide (2014) as a heuristic, error feedback (EF) is a ...
research
∙
09/11/2021
Doubly Adaptive Scaled Algorithm for Machine Learning Using Second-Order Information
We present a novel adaptive optimization algorithm for large-scale machi...
research
∙
08/10/2021
FedPAGE: A Fast Local Stochastic Gradient Method for Communication-Efficient Federated Learning
Federated Averaging (FedAvg, also known as Local-SGD) (McMahan et al., 2...
research
∙
07/20/2021
CANITA: Faster Rates for Distributed Convex Optimization with Communication Compression
Due to the high communication cost in distributed and federated learning...
research
∙
06/09/2021
EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedback
Error feedback (EF), also known as error compensation, is an immensely p...
research
∙
06/08/2021
Lower Bounds and Optimal Algorithms for Smooth and Strongly Convex Decentralized Optimization Over Time-Varying Networks
We consider the task of minimizing the sum of smooth and strongly convex...
research
∙
06/07/2021
Smoothness-Aware Quantization Techniques
Distributed machine learning has become an indispensable tool for traini...
research
∙
06/06/2021
Complexity Analysis of Stein Variational Gradient Descent Under Talagrand's Inequality T1
We study the complexity of Stein Variational Gradient Descent (SVGD), wh...
research
∙
06/06/2021
MURANA: A Generic Framework for Stochastic Variance-Reduced Optimization
We propose a generic variance-reduced algorithm, which we call MUltiple ...
research
∙
06/05/2021
FedNL: Making Newton-Type Methods Applicable to Federated Learning
Inspired by recent work of Islamov et al (2021), we propose a family of ...
research
∙
04/19/2021