Log In Sign Up

Multilevel Minimization for Deep Residual Networks

by   Lisa Gaedke-Merzhäuser, et al.

We present a new multilevel minimization framework for the training of deep residual networks (ResNets), which has the potential to significantly reduce training time and effort. Our framework is based on the dynamical system's viewpoint, which formulates a ResNet as the discretization of an initial value problem. The training process is then formulated as a time-dependent optimal control problem, which we discretize using different time-discretization parameters, eventually generating multilevel-hierarchy of auxiliary networks with different resolutions. The training of the original ResNet is then enhanced by training the auxiliary networks with reduced resolutions. By design, our framework is conveniently independent of the choice of the training strategy chosen on each level of the multilevel hierarchy. By means of numerical examples, we analyze the convergence behavior of the proposed method and demonstrate its robustness. For our examples we employ a multilevel gradient-based methods. Comparisons with standard single level methods show a speedup of more than factor three while achieving the same validation accuracy.


Layer-Parallel Training of Deep Residual Neural Networks

Residual neural networks (ResNets) are a promising class of deep neural ...

Multilevel Initialization for Layer-Parallel Deep Neural Network Training

This paper investigates multilevel initialization strategies for trainin...

A Multilevel Approach to Training

We propose a novel training method based on nonlinear multilevel minimiz...

A multilevel reinforcement learning framework for PDE based control

Reinforcement learning (RL) is a promising method to solve control probl...

Multilevel Active-Set Trust-Region (MASTR) Method for Bound Constrained Minimization

We introduce a novel variant of the recursive multilevel trust-region (R...

BPX Preconditioners for Isogeometric Analysis Using (Truncated) Hierarchical B-splines

We present the construction of additive multilevel preconditioners, also...