Generalization of an Upper Bound on the Number of Nodes Needed to Achieve Linear Separability

02/10/2018
by   Marjolein Troost, et al.
0

An important issue in neural network research is how to choose the number of nodes and layers such as to solve a classification problem. We provide new intuitions based on earlier results by An et al. (2015) by deriving an upper bound on the number of nodes in networks with two hidden layers such that linear separability can be achieved. Concretely, we show that if the data can be described in terms of N finite sets and the used activation function f is non-constant, increasing and has a left asymptote, we can derive how many nodes are needed to linearly separate these sets. This will be an upper bound that depends on the structure of the data. This structure can be analyzed using an algorithm. For the leaky rectified linear activation function, we prove separately that under some conditions on the slope, the same number of layers and nodes as for the aforementioned activation functions is sufficient. We empirically validate our claims.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/04/2022

Understanding Sinusoidal Neural Networks

In this work, we investigate the representation capacity of multilayer p...
research
03/13/2023

Bayes Complexity of Learners vs Overfitting

We introduce a new notion of complexity of functions and we show that it...
research
06/22/2022

Concentration inequalities and optimal number of layers for stochastic deep neural networks

We state concentration and martingale inequalities for the output of the...
research
06/16/2020

Measuring Model Complexity of Neural Networks with Curve Activation Functions

It is fundamental to measure model complexity of deep neural networks. T...
research
08/20/2017

A Capacity Scaling Law for Artificial Neural Networks

By assuming an ideal neural network with gating functions handling the w...
research
06/13/2019

Neural Networks on Groups

Recent work on neural networks has shown that allowing them to build int...
research
02/27/2023

Invariant Layers for Graphs with Nodes of Different Types

Neural networks that satisfy invariance with respect to input permutatio...

Please sign up or login with your details

Forgot password? Click here to reset