DeepAI AI Chat
Log In Sign Up

A Spline Theory of Deep Networks (Extended Version)

by   Randall Balestriero, et al.
Rice University

We build a rigorous bridge between deep networks (DNs) and approximation theory via spline functions and operators. Our key result is that a large class of DNs can be written as a composition of max-affine spline operators (MASOs), which provide a powerful portal through which to view and analyze their inner workings. For instance, conditioned on the input signal, the output of a MASO DN can be written as a simple affine transformation of the input. This implies that a DN constructs a set of signal-dependent, class-specific templates against which the signal is compared via a simple inner product; we explore the links to the classical theory of optimal classification via matched filters and the effects of data memorization. Going further, we propose a simple penalty term that can be added to the cost function of any DN learning algorithm to force the templates to be orthogonal with each other; this leads to significantly improved classifi- cation performance and reduced overfitting with no change to the DN architecture. The spline partition of the input signal space that is implicitly induced by a MASO directly links DNs to the theory of vector quantization (VQ) and K-means clustering, which opens up new geometric avenue to study how DNs organize signals in a hierarchical fashion. To validate the utility of the VQ interpretation, we develop and validate a new distance metric for signals and images that quantifies the difference between their VQ encodings. (This paper is a significantly expanded version of a paper with the same title that will appear at ICML 2018.)


page 9

page 15

page 16

page 17

page 18

page 19

page 33

page 34


The Geometry of Deep Networks: Power Diagram Subdivision

We study the geometry of deep (neural) networks (DNs) with piecewise aff...

Max-Affine Spline Insights into Deep Generative Networks

We connect a large class of Generative Deep Networks (GDNs) with spline ...

Spline quadrature and semi-classical orthogonal Jacobi polynomials

A theory of spline quadrature rules for arbitrary continuity class in a ...

Batch Normalization Explained

A critically important, ubiquitous, and yet poorly understood ingredient...

Max-Affine Spline Insights Into Deep Network Pruning

In this paper, we study the importance of pruning in Deep Networks (DNs)...