Coin Flipping Neural Networks

06/18/2022
by   Yuval Sieradzki, et al.
1

We show that neural networks with access to randomness can outperform deterministic networks by using amplification. We call such networks Coin-Flipping Neural Networks, or CFNNs. We show that a CFNN can approximate the indicator of a d-dimensional ball to arbitrary accuracy with only 2 layers and 𝒪(1) neurons, where a 2-layer deterministic network was shown to require Ω(e^d) neurons, an exponential improvement (arXiv:1610.09887 [cs.LG]). We prove a highly non-trivial result, that for almost any classification problem, there exists a trivially simple network that solves it given a sufficiently powerful generator for the network's weights. Combining these results we conjecture that for most classification problems, there is a CFNN which solves them with higher accuracy or fewer neurons than any deterministic network. Finally, we verify our proofs experimentally using novel CFNN architectures on CIFAR10 and CIFAR100, reaching an improvement of 9.25% from the baseline.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/04/2020

Network size and weights size for memorization with two-layers neural networks

In 1988, Eric B. Baum showed that two-layers neural networks with thresh...
research
06/14/2021

An Exponential Improvement on the Memorization Capacity of Deep Threshold Networks

It is well known that modern deep neural networks are powerful enough to...
research
11/18/2016

NoiseOut: A Simple Way to Prune Neural Networks

Neural networks are usually over-parameterized with significant redundan...
research
02/27/2019

Counting to Ten with Two Fingers: Compressed Counting with Spiking Neurons

We consider the task of measuring time with probabilistic threshold gate...
research
02/07/2019

Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks

Can multilayer neural networks -- typically constructed as highly comple...
research
05/09/2023

Simplicial Hopfield networks

Hopfield networks are artificial neural networks which store memory patt...
research
07/12/2003

Automatic Classification using Self-Organising Neural Networks in Astrophysical Experiments

Self-Organising Maps (SOMs) are effective tools in classification proble...

Please sign up or login with your details

Forgot password? Click here to reset