Adaptive Gradient Quantization for Data-Parallel SGD

10/23/2020 ∙ by Fartash Faghri, et al. ∙ 0 ∙ share

Many communication-efficient variants of SGD use gradient quantization schemes. These schemes are often heuristic and fixed over the course of training. We empirically observe that the statistics of gradients of deep models change during the training. Motivated by this observation, we introduce two adaptive quantization schemes, ALQ and AMQ. In both schemes, processors update their compression schemes in parallel by efficiently computing sufficient statistics of a parametric distribution. We improve the validation accuracy by almost 2 communication setups. Our adaptive methods are also significantly more robust to the choice of hyperparameters.

READ FULL TEXT

NUQSGD: Improved Communication Efficiency for Data-parallel SGD via Nonuniform Quantization

As the size and complexity of models and datasets grow, so does the need...
08/16/2019 ∙ by Ali Ramezani-Kebrya, et al. ∙ 4 ∙ share

read it
Adaptive Quantization of Model Updates for Communication-Efficient Federated Learning

Communication of model updates between client nodes and the central aggr...
02/08/2021 ∙ by Divyansh Jhunjhunwala, et al. ∙ 0 ∙ share

read it
Optimal Gradient Quantization Condition for Communication-Efficient Distributed Training

The communication of gradients is costly for training deep neural networ...
02/25/2020 ∙ by An Xu, et al. ∙ 0 ∙ share

read it
Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computations

Communication bottleneck has been identified as a significant issue in d...
06/06/2019 ∙ by Debraj Basu, et al. ∙ 0 ∙ share

read it
On Efficient Constructions of Checkpoints

Efficient construction of checkpoints/snapshots is a critical tool for t...
09/28/2020 ∙ by Yu Chen, et al. ∙ 5 ∙ share

read it
PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization

We study gradient compression methods to alleviate the communication bot...
05/31/2019 ∙ by Thijs Vogels, et al. ∙ 0 ∙ share

read it
MQGrad: Reinforcement Learning of Gradient Quantization in Parameter Server

One of the most significant bottleneck in training large scale machine l...
04/22/2018 ∙ by Guoxin Cui, et al. ∙ 0 ∙ share

read it

Adaptive Gradient Quantization for Data-Parallel SGD

Authors

NUQSGD: Improved Communication Efficiency for Data-parallel SGD via Nonuniform Quantization

Adaptive Quantization of Model Updates for Communication-Efficient Federated Learning

Optimal Gradient Quantization Condition for Communication-Efficient Distributed Training

Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computations

On Efficient Constructions of Checkpoints

PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization

MQGrad: Reinforcement Learning of Gradient Quantization in Parameter Server

This week in AI