Trainable Activation Function in Image Classification

04/28/2020
by   Zhaohe Liao, et al.
0

In the current research of neural networks, the activation function is manually specified by human and not able to change themselves during training. This paper focus on how to make the activation function trainable for deep neural networks. We use series and linear combination of different activation functions make activation functions continuously variable. Also, we test the performance of CNNs with Fourier series simulated activation(Fourier-CNN) and CNNs with linear combined activation function (LC-CNN) on Cifar-10 dataset. The result shows our trainable activation function reveals better performance than the most used ReLU activation function. Finally, we improves the performance of Fourier-CNN with Autoencoder, and test the performance of PSO algorithm in optimizing the parameters of networks

READ FULL TEXT
research
04/28/2020

Trainable Activation Function Supported CNN in Image Classification

In the current research of neural networks, the activation function is m...
research
05/26/2019

ProbAct: A Probabilistic Activation Function for Deep Neural Networks

Activation functions play an important role in the training of artificia...
research
02/08/2019

A simple and efficient architecture for trainable activation functions

Learning automatically the best activation function for the task is an a...
research
02/08/2019

Fourier Neural Networks: A Comparative Study

We review neural network architectures which were motivated by Fourier s...
research
01/09/2020

On the loss of learning capability inside an arrangement of neural networks

We analyze the loss of information and the loss of learning capability i...
research
06/13/2023

Safe Use of Neural Networks

Neural networks in modern communication systems can be susceptible to in...
research
11/15/2022

Characterizing the Spectrum of the NTK via a Power Series Expansion

Under mild conditions on the network initialization we derive a power se...

Please sign up or login with your details

Forgot password? Click here to reset