Deep Learning is Singular, and That's Good

by   Daniel Murfet, et al.

In singular models, the optimal set of parameters forms an analytic set with singularities and classical statistical inference cannot be applied to such models. This is significant for deep learning as neural networks are singular and thus "dividing" by the determinant of the Hessian or employing the Laplace approximation are not appropriate. Despite its potential for addressing fundamental issues in deep learning, singular learning theory appears to have made little inroads into the developing canon of deep learning theory. Via a mix of theory and experiment, we present an invitation to singular learning theory as a vehicle for understanding deep learning and suggest important future work to make singular learning theory directly applicable to how deep learning is performed in practice.



There are no comments yet.


page 9


Singular matrix variate Birnbaum-Saunders distribution under elliptical models

This work sets the matrix variate Birnbaum-Saunders theory in the contex...

Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice

It is well known that the initialization of weights in deep neural netwo...

The dually flat structure for singular models

The dually flat structure introduced by Amari-Nagaoka is highlighted in ...

Lightlike Neuromanifolds, Occam's Razor and Deep Learning

Why do deep neural networks generalize with a very high dimensional para...

Quadratic Suffices for Over-parametrization via Matrix Chernoff Bound

We improve the over-parametrization size over two beautiful results [Li ...

An algorithm to determine regular singular Mahler systems

This paper is devoted to the study of the analytic properties of Mahler ...

On the existence of four or more curved foldings with common creases and crease patterns

Consider a curve Γ in a domain D in the plane R^2. Thinking of D as a pi...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.