Lon-eå at SemEval-2023 Task 11: A Comparison of Activation Functions for Soft and Hard Label Prediction

by   Peyman Hosseini, et al.
Luleå University of Technology
Imperial College London

We study the influence of different activation functions in the output layer of deep neural network models for soft and hard label prediction in the learning with disagreement task. In this task, the goal is to quantify the amount of disagreement via predicting soft labels. To predict the soft labels, we use BERT-based preprocessors and encoders and vary the activation function used in the output layer, while keeping other parameters constant. The soft labels are then used for the hard label prediction. The activation functions considered are sigmoid as well as a step-function that is added to the model post-training and a sinusoidal activation function, which is introduced for the first time in this paper.


page 1

page 2

page 3

page 4


Activation Adaptation in Neural Networks

Many neural network architectures rely on the choice of the activation f...

Activation functions are not needed: the ratio net

The function approximator that finds the function mapping the feature to...

PHN: Parallel heterogeneous network with soft gating for CTR prediction

The Click-though Rate (CTR) prediction task is a basic task in recommend...

Saturating Auto-Encoders

We introduce a simple new regularizer for auto-encoders whose hidden-uni...

Reservoir Computing and its Sensitivity to Symmetry in the Activation Function

Reservoir computing has repeatedly been shown to be extremely successful...

Transformers with Learnable Activation Functions

Activation functions can have a significant impact on reducing the topol...

Effect of the output activation function on the probabilities and errors in medical image segmentation

The sigmoid activation is the standard output activation function in bin...

Please sign up or login with your details

Forgot password? Click here to reset