Deep neural networks are biased towards simple functions

12/25/2018
by   Giacomo De Palma, et al.
0

We prove that the binary classifiers of bit strings generated by random wide deep neural networks are biased towards simple functions. The simplicity is captured by the following two properties. For any given input bit string, the average Hamming distance of the closest input bit string with a different classification is at least √(n/(2π n).), where n is the length of the string. Moreover, if the bits of the initial string are flipped randomly, the average number of flips required to change the classification grows linearly with n. On the contrary, for a uniformly random binary classifier, the average Hamming distance of the closest input bit string with a different classification is one, and the average number of random flips required to change the classification is two. These results are confirmed by numerical experiments on deep neural networks with two hidden layers, and settle the conjecture stating that random deep neural networks are biased towards simple functions. The conjecture that random deep neural networks are biased towards simple functions was proposed and numerically explored in [Valle Pérez et al., arXiv:1805.08522] to explain the unreasonably good generalization properties of deep learning algorithms. By providing a precise characterization of the form of this bias towards simplicity, our results open the way to a rigorous proof of the generalization properties of deep learning algorithms in real-world scenarios.

READ FULL TEXT
research
05/22/2018

Deep learning generalizes because the parameter-function map is biased towards simple functions

Deep neural networks generalize remarkably well without explicit regular...
research
04/13/2020

Adversarial robustness guarantees for random deep neural networks

The reliability of most deep learning algorithms is fundamentally challe...
research
01/30/2019

A Simple Explanation for the Existence of Adversarial Examples with Small Hamming Distance

The existence of adversarial examples in which an imperceptible change i...
research
04/22/2016

K-Bit-Swap: A New Operator For Real-Coded Evolutionary Algorithms

There has been a variety of crossover operators proposed for Real-Coded ...
research
01/13/2018

Longest Common Prefixes with k-Errors and Applications

Although real-world text datasets, such as DNA sequences, are far from b...
research
02/04/2020

Faster Binary Mean Computation Under Dynamic Time Warping

Many consensus string problems are based on Hamming distance. We replace...
research
02/16/2019

Cost vs. Information Tradeoffs for Treasure Hunt in the Plane

A mobile agent has to find an inert treasure hidden in the plane. Both t...

Please sign up or login with your details

Forgot password? Click here to reset