A Survey on Activation Functions and their relation with Xavier and He Normal Initialization

03/18/2020
by   Leonid Datta, et al.
0

In artificial neural network, the activation function and the weight initialization method play important roles in training and performance of a neural network. The question arises is what properties of a function are important/necessary for being a well-performing activation function. Also, the most widely used weight initialization methods - Xavier and He normal initialization have fundamental connection with activation function. This survey discusses the important/necessary properties of activation function and the most widely used activation functions (sigmoid, tanh, ReLU, LReLU and PReLU). This survey also explores the relationship between these activation functions and the two weight initialization methods - Xavier and He normal initialization.

READ FULL TEXT
research
07/31/2023

STL: A Signed and Truncated Logarithm Activation Function for Neural Networks

Activation functions play an essential role in neural networks. They pro...
research
01/05/2020

Cooperative Initialization based Deep Neural Network Training

Researchers have proposed various activation functions. These activation...
research
05/24/2022

Imposing Gaussian Pre-Activations in a Neural Network

The goal of the present work is to propose a way to modify both the init...
research
09/24/2018

Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function

We demonstrate that in residual neural networks (ResNets) dynamical isom...
research
09/18/2021

AutoInit: Analytic Signal-Preserving Weight Initialization for Neural Networks

Neural networks require careful weight initialization to prevent signals...
research
07/31/2020

Neural Network Degeneration and its Relationship to the Brain

This report discusses the application of neural networks (NNs) as small ...
research
10/12/2020

Activation function impact on Sparse Neural Networks

While the concept of a Sparse Neural Network has been researched for som...

Please sign up or login with your details

Forgot password? Click here to reset