Learning activation functions from data using cubic spline interpolation

05/18/2016
by   Simone Scardapane, et al.
0

Neural networks require a careful design in order to perform properly on a given task. In particular, selecting a good activation function (possibly in a data-dependent fashion) is a crucial step, which remains an open problem in the research community. Despite a large amount of investigations, most current implementations simply select one fixed function from a small set of candidates, which is not adapted during training, and is shared among all neurons throughout the different layers. However, neither two of these assumptions can be supposed optimal in practice. In this paper, we present a principled way to have data-dependent adaptation of the activation functions, which is performed independently for each neuron. This is achieved by leveraging over past and present advances on cubic spline interpolation, allowing for local adaptation of the functions around their regions of use. The resulting algorithm is relatively cheap to implement, and overfitting is counterbalanced by the inclusion of a novel damping criterion, which penalizes unwanted oscillations from a predefined shape. Experimental results validate the proposal over two well-known benchmarks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/28/2019

Activation Adaptation in Neural Networks

Many neural network architectures rely on the choice of the activation f...
research
05/02/2020

A survey on modern trainable activation functions

In the literature, there is a strong interest to identify and define act...
research
02/06/2019

Widely Linear Kernels for Complex-Valued Kernel Activation Functions

Complex-valued neural networks (CVNNs) have been shown to be powerful no...
research
07/13/2017

Kafnets: kernel-based non-parametric activation functions for neural networks

Neural networks are generally built by interleaving (adaptable) linear l...
research
06/22/2020

Advantages of biologically-inspired adaptive neural activation in RNNs during learning

Dynamic adaptation in single-neuron response plays a fundamental role in...
research
10/15/2021

Construction of C^2 cubic splines on arbitrary triangulations

In this paper, we address the problem of constructing C^2 cubic spline f...

Please sign up or login with your details

Forgot password? Click here to reset