Activation function dependence of the storage capacity of treelike neural networks

07/21/2020
by   Jacob A. Zavatone-Veth, et al.
0

The expressive power of artificial neural networks crucially depends on the nonlinearity of their activation functions. Though a wide variety of nonlinear activation functions have been proposed for use in artificial neural networks, a detailed understanding of their role in determining the expressive power of a network has not emerged. Here, we study how activation functions affect the storage capacity of treelike two-layer networks. We relate the boundedness or divergence of the capacity in this limit to the smoothness of the activation function, elucidating the relationship between previously studied special cases. Our results show that nonlinearity can both increase capacity and decrease the robustness of classification, and provide simple estimates for the capacity of networks with several commonly used activation functions.

READ FULL TEXT
research
07/13/2023

Deep Network Approximation: Beyond ReLU to Diverse Activation Functions

This paper explores the expressive power of deep neural networks for a d...
research
02/02/2021

Formalising the Use of the Activation Function in Neural Inference

We investigate how activation functions can be used to describe neural f...
research
10/22/2021

Logical Activation Functions: Logit-space equivalents of Boolean Operators

Neuronal representations within artificial neural networks are commonly ...
research
03/17/2022

On the expressive power of message-passing neural networks as global feature map transformers

We investigate the power of message-passing neural networks (MPNNs) in t...
research
07/31/2022

Functional Rule Extraction Method for Artificial Neural Networks

The idea I propose in this paper is a method that is based on comprehens...
research
04/29/2022

Wide and Deep Neural Networks Achieve Optimality for Classification

While neural networks are used for classification tasks across domains, ...
research
05/01/2023

Activation Functions Not To Active: A Plausible Theory on Interpreting Neural Networks

Researchers commonly believe that neural networks model a high-dimension...

Please sign up or login with your details

Forgot password? Click here to reset