Deep Learning is Singular, and That's Good

10/22/2020
by   Daniel Murfet, et al.
0

In singular models, the optimal set of parameters forms an analytic set with singularities and classical statistical inference cannot be applied to such models. This is significant for deep learning as neural networks are singular and thus "dividing" by the determinant of the Hessian or employing the Laplace approximation are not appropriate. Despite its potential for addressing fundamental issues in deep learning, singular learning theory appears to have made little inroads into the developing canon of deep learning theory. Via a mix of theory and experiment, we present an invitation to singular learning theory as a vehicle for understanding deep learning and suggest important future work to make singular learning theory directly applicable to how deep learning is performed in practice.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 9

12/19/2019

Singular matrix variate Birnbaum-Saunders distribution under elliptical models

This work sets the matrix variate Birnbaum-Saunders theory in the contex...
11/13/2017

Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice

It is well known that the initialization of weights in deep neural netwo...
11/23/2020

The dually flat structure for singular models

The dually flat structure introduced by Amari-Nagaoka is highlighted in ...
05/27/2019

Lightlike Neuromanifolds, Occam's Razor and Deep Learning

Why do deep neural networks generalize with a very high dimensional para...
06/09/2019

Quadratic Suffices for Over-parametrization via Matrix Chernoff Bound

We improve the over-parametrization size over two beautiful results [Li ...
02/22/2021

An algorithm to determine regular singular Mahler systems

This paper is devoted to the study of the analytic properties of Mahler ...
11/17/2019

On the existence of four or more curved foldings with common creases and crease patterns

Consider a curve Γ in a domain D in the plane R^2. Thinking of D as a pi...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.