ResIST: Layer-Wise Decomposition of ResNets for Distributed Training

07/02/2021
by   Chen Dun, et al.
0

We propose , a novel distributed training protocol for Residual Networks (ResNets). randomly decomposes a global ResNet into several shallow sub-ResNets that are trained independently in a distributed manner for several local iterations, before having their updates synchronized and aggregated into the global model. In the next round, new sub-ResNets are randomly generated and the process repeats. By construction, per iteration, communicates only a small portion of network parameters to each machine and never uses the full model during training. Thus, reduces the communication, memory, and time requirements of ResNet training to only a fraction of the requirements of previous methods. In comparison to common protocols like data-parallel training and data-parallel training with local SGD, yields a decrease in wall-clock training time, while being competitive with respect to model performance.

READ FULL TEXT

page 1

page 3

research
02/20/2021

GIST: Distributed Training for Large-Scale Graph Convolutional Networks

The graph convolutional network (GCN) is a go-to solution for machine le...
research
09/06/2020

HLSGD Hierarchical Local SGD With Stale Gradients Featuring

While distributed training significantly speeds up the training process ...
research
04/12/2021

Accelerating Neural Network Training with Distributed Asynchronous and Selective Optimization (DASO)

With increasing data and model complexities, the time required to train ...
research
10/19/2018

Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD

Large-scale machine learning training, in particular distributed stochas...
research
11/08/2018

Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training

Distributed training of deep nets is an important technique to address s...
research
11/10/2015

Reducing the Training Time of Neural Networks by Partitioning

This paper presents a new method for pre-training neural networks that c...

Please sign up or login with your details

Forgot password? Click here to reset