AI Chat AI Image Generator AI Video Text to Speech

Negative eigenvalues of the Hessian in deep neural networks

02/06/2019

∙

by Guillaume Alain, et al.

∙

∙

The loss function of deep networks is known to be non-convex but the precise nature of this nonconvexity is still an active area of research. In this work, we study the loss landscape of deep networks through the eigendecompositions of their Hessian matrix. In particular, we examine how important the negative eigenvalues are and the benefits one can observe in handling them appropriately.

Guillaume Alain
11 publications
Nicolas Le Roux
23 publications
Pierre-Antoine Manzagol
5 publications

page 1

page 2

page 3

page 4

research

∙ 11/22/2016

Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond

We look at the eigenvalues of the Hessian of a loss function before and ...

0 Levent Sagun, et al. ∙

research

∙ 01/14/2020

On the Convex Behavior of Deep Neural Networks in Relation to the Layers' Width

The Hessian of neural networks can be decomposed into a sum of two matri...

0 Etai Littwin, et al. ∙

research

∙ 10/27/2021

Does the Data Induce Capacity Control in Deep Learning?

This paper studies how the dataset may be the cause of the anomalous gen...

0 Yang Rubing, et al. ∙

research

∙ 03/02/2021

Hessian Eigenspectra of More Realistic Nonlinear Models

Given an optimization problem, the Hessian matrix and its eigenspectrum ...

0 Zhenyu Liao, et al. ∙

research

∙ 11/05/2016

Loss-aware Binarization of Deep Networks

Deep neural network models, though very powerful and highly successful, ...

0 Lu Hou, et al. ∙

research

∙ 07/11/2019

The Landscape of Non-convex Empirical Risk with Degenerate Population Risk

The landscape of empirical risk has been widely studied in a series of m...

0 Shuang Li, et al. ∙

research

∙ 07/06/2018

The Goldilocks zone: Towards better understanding of neural network loss landscapes

We explore the loss landscape of fully-connected neural networks using r...

0 Stanislav Fort, et al. ∙