Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation

05/29/2021
by   Mikhail Belkin, et al.
0

In the past decade the mathematical theory of machine learning has lagged far behind the triumphs of deep neural networks on practical challenges. However, the gap between theory and practice is gradually starting to close. In this paper I will attempt to assemble some pieces of the remarkable and still incomplete mathematical mosaic emerging from the efforts to understand the foundations of deep learning. The two key themes will be interpolation, and its sibling, over-parameterization. Interpolation corresponds to fitting data, even noisy data, exactly. Over-parameterization enables interpolation and provides flexibility to select a right interpolating model. As we will see, just as a physical prism separates colors mixed within a ray of light, the figurative prism of interpolation helps to disentangle generalization and optimization properties within the complex picture of modern Machine Learning. This article is written with belief and hope that clearer understanding of these issues brings us a step closer toward a general theory of deep learning and machine learning.

READ FULL TEXT
research
06/19/2019

Transfinite mean value interpolation over polygons

Mean value interpolation is a method for fitting a smooth function to pi...
research
12/28/2018

Reconciling modern machine learning and the bias-variance trade-off

The question of generalization in machine learning---how algorithms are ...
research
07/20/2020

DeepNNK: Explaining deep models and their generalization using polytope interpolation

Modern machine learning systems based on neural networks have shown grea...
research
03/31/2021

Fitting Elephants

Textbook wisdom advocates for smooth function fits and implies that inte...
research
03/25/2018

Goldbach's Function Approximation Using Deep Learning

Goldbach conjecture is one of the most famous open mathematical problems...
research
04/15/2015

Bridging belief function theory to modern machine learning

Machine learning is a quickly evolving field which now looks really diff...

Please sign up or login with your details

Forgot password? Click here to reset