AI Chat AI Image Generator AI Video Text to Speech

Distributed Training of Deep Neural Networks: Theoretical and Practical Limits of Parallel Scalability

09/22/2016

∙

by Janis Keuper, et al.

∙

∙

This paper presents a theoretical analysis and practical evaluation of the main bottlenecks towards a scalable distributed solution for the training of Deep Neuronal Networks (DNNs). The presented results show, that the current state of the art approach, using data-parallelized Stochastic Gradient Descent (SGD), is quickly turning into a vastly communication bound problem. In addition, we present simple but fixed theoretic constraints, preventing effective scaling of DNN training beyond only a few dozen nodes. This leads to poor scalability of DNN training in most practical scenarios.

Janis Keuper
19 publications
Franz-Josef Pfreundt
13 publications

page 2

page 4

research

∙ 01/24/2018

On Scale-out Deep Learning Training for Cloud and HPC

The exponential growth in use of large deep neural networks has accelera...

0 Srinivas Sridharan, et al. ∙

research

∙ 12/09/2015

Distributed Training of Deep Neural Networks with Theoretical Analysis: Under SSP Setting

We propose a distributed approach to train deep neural networks (DNNs), ...

0 Abhimanu Kumar, et al. ∙

research

∙ 02/26/2018

Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis

Deep Neural Networks (DNNs) are becoming an important tool in modern com...

0 Tal Ben-Nun, et al. ∙

research

∙ 04/23/2021

Partitioning sparse deep neural networks for scalable training and inference

The state-of-the-art deep neural networks (DNNs) have significant comput...

0 Gunduz Vehbi Demirci, et al. ∙

research

∙ 06/25/2018

Pushing the boundaries of parallel Deep Learning -- A practical approach

This work aims to assess the state of the art of data parallel deep neur...

0 Paolo Viviani, et al. ∙

research

∙ 06/01/2017

CATERPILLAR: Coarse Grain Reconfigurable Architecture for Accelerating the Training of Deep Neural Networks

Accelerating the inference of a trained DNN is a well studied subject. I...

0 Yuanfang Li, et al. ∙

research

∙ 01/04/2023

Scalable Optimal Design of Incremental Volt/VAR Control using Deep Neural Networks

Volt/VAR control rules facilitate the autonomous operation of distribute...

0 Sarthak Gupta, et al. ∙