Layer-Parallel Training with GPU Concurrency of Deep Residual Neural Networks via Nonlinear Multigrid

07/14/2020

∙

A Multigrid Full Approximation Storage algorithm for solving Deep Residual Networks is developed to enable neural network parallelized layer-wise training and concurrent computational kernel execution on GPUs. This work demonstrates a 10.2x speedup over traditional layer-wise model parallelism techniques using the same number of compute units.

READ FULL TEXT

Layer-Parallel Training with GPU Concurrency of Deep Residual Neural Networks via Nonlinear Multigrid

Layer-Parallel Training of Deep Residual Neural Networks

PARTIME: Scalable and Parallel Processing Over Time with Deep Neural Networks

Brief Announcement: On the Limits of Parallelizing Convolutional Neural Networks on GPUs

Penalty and Augmented Lagrangian Methods for Layer-parallel Training of Residual Networks

Neural Networks as Paths through the Space of Representations

Exploiting Sparsity in Pruned Neural Networks to Optimize Large Model Training

Deep Layer-wise Networks Have Closed-Form Weights

Layer-Parallel Training with GPU Concurrency of Deep Residual Neural Networks via Nonlinear Multigrid

Related Research

Layer-Parallel Training of Deep Residual Neural Networks

PARTIME: Scalable and Parallel Processing Over Time with Deep Neural Networks

Brief Announcement: On the Limits of Parallelizing Convolutional Neural Networks on GPUs

Penalty and Augmented Lagrangian Methods for Layer-parallel Training of Residual Networks

Neural Networks as Paths through the Space of Representations

Exploiting Sparsity in Pruned Neural Networks to Optimize Large Model Training

Deep Layer-wise Networks Have Closed-Form Weights