Scaling Distributed Training of Flood-Filling Networks on HPC Infrastructure for Brain Mapping

05/13/2019

∙

Mapping all the neurons in the brain requires automatic reconstruction of entire cells from volume electron microscopy data. The flood-filling networks (FFN) architecture can achieve leading performance. However, the training of the network is computationally very expensive. In order to reduce the training time, we implemented synchronous and data-parallel distributed training using the Horovod framework on top of the published FFN code. We demonstrated the scaling of FFN training up to 1024 Intel Knights Landing (KNL) nodes at Argonne Leadership Computing Facility. We investigated the training accuracy with different optimizers, learning rates, and optional warm-up periods. We discovered that square root scaling for learning rate works best beyond 16 nodes, which is contrary to the case of smaller number of nodes, where linear learning rate scaling with warm-up performs the best. Our distributed training reaches 95 optimizer.

READ FULL TEXT

Scaling Distributed Training of Flood-Filling Networks on HPC Infrastructure for Brain Mapping

CProp: Adaptive Learning Rate Scaling from Past Gradient Conformity

Unintended Effects on Adaptive Learning Rate for Training Neural Network with Output Scale Change

On Scale-out Deep Learning Training for Cloud and HPC

To Raise or Not To Raise: The Autonomous Learning Rate Question

Training Distributed Deep Recurrent Neural Networks with Mixed Precision on GPU Clusters

Training With Data Dependent Dynamic Learning Rates

Accurate, Fast and Scalable Kernel Ridge Regression on Parallel and Distributed Systems

Scaling Distributed Training of Flood-Filling Networks on HPC Infrastructure for Brain Mapping

Related Research

CProp: Adaptive Learning Rate Scaling from Past Gradient Conformity

Unintended Effects on Adaptive Learning Rate for Training Neural Network with Output Scale Change

On Scale-out Deep Learning Training for Cloud and HPC

To Raise or Not To Raise: The Autonomous Learning Rate Question

Training Distributed Deep Recurrent Neural Networks with Mixed Precision on GPU Clusters

Training With Data Dependent Dynamic Learning Rates

Accurate, Fast and Scalable Kernel Ridge Regression on Parallel and Distributed Systems