Rosenblatt's first theorem and frugality of deep learning

08/29/2022
by   A. N. Kirdin, et al.
0

First Rosenblatt's theorem about omnipotence of shallow networks states that elementary perceptrons can solve any classification problem if there are no discrepancies in the training set. Minsky and Papert considered elementary perceptrons with restrictions on the neural inputs: a bounded number of connections or a relatively small diameter of the receptive field for each neuron at the hidden layer. They proved that under these constraints, an elementary perceptron cannot solve some problems, such as the connectivity of input images or the parity of pixels in them. In this note, we demonstrated first Rosenblatt's theorem at work, showed how an elementary perceptron can solve a version of the travel maze problem, and analysed the complexity of that solution. We constructed also a deep network algorithm for the same problem. It is much more efficient. The shallow network uses an exponentially large number of neurons on the hidden layer (Rosenblatt's A-elements), whereas for the deep network the second order polynomial complexity is sufficient. We demonstrated that for the same complex problem deep network can be much smaller and reveal a heuristic behind this effect.

READ FULL TEXT
research
09/27/2015

Representation Benefits of Deep Feedforward Networks

This note provides a family of classification problems, indexed by a pos...
research
03/06/2017

On the Expressive Power of Overlapping Architectures of Deep Learning

Expressive efficiency refers to the relation between two architectures A...
research
12/20/2013

On the number of response regions of deep feed forward networks with piece-wise linear activations

This paper explores the complexity of deep feedforward networks with lin...
research
03/25/2022

Qualitative neural network approximation over R and C: Elementary proofs for analytic and polynomial activation

In this article, we prove approximation theorems in classes of deep and ...
research
01/15/2013

Pushing Stochastic Gradient towards Second-Order Methods -- Backpropagation Learning with Transformations in Nonlinearities

Recently, we proposed to transform the outputs of each hidden neuron in ...
research
12/14/2020

Perceptron Theory for Predicting the Accuracy of Neural Networks

Many neural network models have been successful at classification proble...
research
02/22/2019

Characterizing PSPACE with shallow non-confluent P systems

In P systems with active membranes, the question of understanding the po...

Please sign up or login with your details

Forgot password? Click here to reset