Deep learning improved by biological activation functions

03/19/2018
by   Gardave S Bhumbra, et al.
0

`Biologically inspired' activation functions, such as the logistic sigmoid, have been instrumental in the historical development of machine learning techniques. However in the field of deep learning, they have been largely displaced by rectified (ReLU) or exponential (ELU) linear units to mitigate the effects of vanishing gradients associated with error back-propagation. The logistic sigmoid however does not represent the true input-output relation in real neurones under physiological conditions. Here, radical root unit (RRU) activation functions are introduced, exhibiting input-output non-linearities that are substantially more biologically plausible since their functional form is based on known current-frequency relationships. In order to evaluate whether RRU activations improve deep learning performance, networks are constructed with identical architectures except differing in their transfer functions (ReLU, ELU, and RRUs). Multilayer perceptrons, stacked auto-encoders, and convolutional networks are used to test supervised and unsupervised learning based on the MNIST dataset. Results of learning performance, quantified using loss and error measurements, demonstrate that the RRU networks not only train faster than their ReLU and ELU counterparts, but also lead to improved generalised models in the absence of formal regularisation. These results therefore confirm that revisiting the properties of biological neurones and their circuitry might prove invaluable in the field of deep learning.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/05/2021

An Analysis of State-of-the-art Activation Functions For Supervised Deep Neural Network

This paper provides an analysis of state-of-the-art activation functions...
research
05/12/2023

Saturated Non-Monotonic Activation Functions

Activation functions are essential to deep learning networks. Popular an...
research
07/25/2019

Fast generalization error bound of deep learning without scale invariance of activation functions

In theoretical analysis of deep learning, discovering which features of ...
research
06/22/2020

Advantages of biologically-inspired adaptive neural activation in RNNs during learning

Dynamic adaptation in single-neuron response plays a fundamental role in...
research
08/06/2020

The nlogistic-sigmoid function

The variants of the logistic-sigmoid functions used in artificial neural...
research
03/05/2018

Conducting Credit Assignment by Aligning Local Representations

The use of back-propagation and its variants to train deep networks is o...
research
02/27/2023

Complex Clipping for Improved Generalization in Machine Learning

For many machine learning applications, a common input representation is...

Please sign up or login with your details

Forgot password? Click here to reset