
Integrated Model, Batch and Domain Parallelism in Training Neural Networks
We propose a new integrated method of exploiting model, batch and domain...
read it

A Linear Algebraic Approach to Model Parallelism in Deep Learning
Training deep neural networks (DNNs) in largecluster computing environm...
read it

Demystifying Parallel and Distributed Deep Learning: An InDepth Concurrency Analysis
Deep Neural Networks (DNNs) are becoming an important tool in modern com...
read it

GPU Asynchronous Stochastic Gradient Descent to Speed Up Neural Network Training
The ability to train largescale neural networks has resulted in stateo...
read it

The Case for Strong Scaling in Deep Learning: Training Large 3D CNNs with Hybrid Parallelism
We present scalable hybridparallel algorithms for training largescale ...
read it

dMath: Distributed Linear Algebra for DL
The paper presents a parallel math library, dMath, that demonstrates lea...
read it

Fast Distributed Training of Deep Neural Networks: Dynamic Communication Thresholding for Model and Data Parallelism
Data Parallelism (DP) and Model Parallelism (MP) are two common paradigm...
read it
Integrated Model and Data Parallelism in Training Neural Networks
We propose a new integrated method of exploiting both model and data parallelism for the training of deep neural networks (DNNs) on large distributedmemory computers using minibatch stochastic gradient descent (SGD). Our goal is to find an efficient parallelization strategy for a fixed batch size using P processes. Our method is inspired by the communicationavoiding algorithms in numerical linear algebra. We see P processes as logically divided into a P_r × P_c grid where the P_r dimension is implicitly responsible for model parallelism and the P_c dimension is implicitly responsible for data parallelism. In practice, the integrated matrixbased parallel algorithm encapsulates both types of parallelism automatically. We analyze the communication complexity and analytically demonstrate that the lowest communication costs are often achieved neither with pure model parallelism nor with pure data parallelism. We also show the positive effect of our approach in the computational performance of SGD based DNN training where the reduced number of processes responsible for data parallelism result in "fatter" matrices that enable higherthroughput matrix multiplication.
READ FULL TEXT
Comments
There are no comments yet.