Comparison of non-linear activation functions for deep neural networks on MNIST classification task

04/08/2018
by   Dabal Pedamonti, et al.
0

Activation functions play a key role in neural networks so it becomes fundamental to understand their advantages and disadvantages in order to achieve better performances. This paper will first introduce common types of non linear activation functions that are alternative to the well known sigmoid function and then evaluate their characteristics. Moreover deeper neural networks will be analysed because they positively influence the final performances compared to shallower networks. They also strictly depend on the weight initialisation hence the effect of drawing weights from Gaussian and uniform distribution will be analysed making particular attention on how the number of incoming and outgoing connection to a node influence the whole network.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/30/2017

Empirical analysis of non-linear activation functions for Deep Neural Networks in classification tasks

We provide an overview of several non-linear activation functions in a n...
research
03/22/2022

Exploring Linear Feature Disentanglement For Neural Networks

Non-linear activation functions, e.g., Sigmoid, ReLU, and Tanh, have ach...
research
03/05/2016

Network Morphism

We present in this paper a systematic study on how to morph a well-train...
research
08/22/2018

Multi-Grained-Attention Gated Convolutional Neural Networks for Sentence Classification

The classification task of sentences is very challenging because of the ...
research
07/15/2020

Attention as Activation

Activation functions and attention mechanisms are typically treated as h...
research
03/22/2023

Fixed points of arbitrarily deep 1-dimensional neural networks

In this paper, we introduce a new class of functions on ℝ that is closed...
research
10/05/2019

Minimum "Norm" Neural Networks are Splines

We develop a general framework based on splines to understand the interp...

Please sign up or login with your details

Forgot password? Click here to reset