Analysis of the Gradient Descent Algorithm for a Deep Neural Network Model with Skip-connections

04/10/2019

∙

The behavior of the gradient descent (GD) algorithm is analyzed for a deep neural network model with skip-connections. It is proved that in the over-parametrized regime, for a suitable initialization, with high probability GD can find a global minimum exponentially fast. Generalization error estimates along the GD path are also established. As a consequence, it is shown that when the target function is in the reproducing kernel Hilbert space (RKHS) with a kernel defined by the initialization, there exist generalizable early-stopping solutions along the GD path. In addition, it is also shown that the GD path is uniformly close to the functions given by the related random feature model. Consequently, in this "implicit regularization" setting, the deep neural network model deteriorates to a random feature model. Our results hold for neural networks of any width larger than the input dimension.

READ FULL TEXT

Analysis of the Gradient Descent Algorithm for a Deep Neural Network Model with Skip-connections

A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics

On the universal consistency of an over-parametrized deep neural network estimate learned by gradient descent

Random feature approximation for general spectral methods

Analysis of the rate of convergence of an over-parametrized deep neural network estimate learned by gradient descent

Gradient Descent in Neural Networks as Sequential Learning in RKBS

Statistical Optimality of Deep Wide Neural Networks

An initial alignment between neural network and target is needed for gradient descent to learn

Analysis of the Gradient Descent Algorithm for a Deep Neural Network Model with Skip-connections

Related Research

A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics

On the universal consistency of an over-parametrized deep neural network estimate learned by gradient descent

Random feature approximation for general spectral methods

Analysis of the rate of convergence of an over-parametrized deep neural network estimate learned by gradient descent

Gradient Descent in Neural Networks as Sequential Learning in RKBS

Statistical Optimality of Deep Wide Neural Networks

An initial alignment between neural network and target is needed for gradient descent to learn