The asymptotic spectrum of the Hessian of DNN throughout training

10/01/2019

∙

The dynamics of DNNs during gradient descent is described by the so-called Neural Tangent Kernel (NTK). In this article, we show that the NTK allows one to gain precise insight into the Hessian of the cost of DNNs: we obtain a full characterization of the asymptotics of the spectrum of the Hessian, at initialization and during training.

READ FULL TEXT

The asymptotic spectrum of the Hessian of DNN throughout training

How noise affects the Hessian spectrum in overparameterized neural networks

The Full Spectrum of Deep Net Hessians At Scale: Dynamics with Sample Size

Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians

PyHessian: Neural Networks Through the Lens of the Hessian

Combining Natural Gradient with Hessian Free Methods for Sequence Training

An Investigation into Neural Net Optimization via Hessian Eigenvalue Density

Hessian and increasing-Hessian orderings of multivariate skew-elliptical random vectors

The asymptotic spectrum of the Hessian of DNN throughout training

Related Research

How noise affects the Hessian spectrum in overparameterized neural networks

The Full Spectrum of Deep Net Hessians At Scale: Dynamics with Sample Size

Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians

PyHessian: Neural Networks Through the Lens of the Hessian

Combining Natural Gradient with Hessian Free Methods for Sequence Training

An Investigation into Neural Net Optimization via Hessian Eigenvalue Density

Hessian and increasing-Hessian orderings of multivariate skew-elliptical random vectors