Why KDAC? A general activation function for knowledge discovery

by   Zhenhua Wang, et al.

Named entity recognition based on deep learning (DNER) can effectively mine expected knowledge from large-scale unstructured and semi-structured text, and has gradually become the paradigm of knowledge discovery. Currently, Tanh, ReLU and Sigmoid dominate DNER, however, these activation functions failed to treat gradient vanishing, no negative output or non-differentiable existence, which may impede DNER's exploration of knowledge caused by the omission and the incomplete representation of latent semantics. To surmount the non-negligible obstacle, we present a novel and general activation function termed KDAC. Detailly, KDAC is a thought that can aggregate and inherit the merits of Tanh and ReLU since they are widely leveraged in various knowledge domains. The positive region corresponds to an adaptive linear design encouraged by ReLU. The negative region considers the interaction between exponent and linearity to surmount the obstacle of gradient vanishing and no negative value. Crucially, the non-differentiable points are alerted and eliminated by a smooth approximation. We perform experiments based on BERT-BiLSTM-CNN-CRF model on six benchmark datasets containing different domain knowledge, such as Weibo, Clinical, E-commerce, Resume, HAZOP and People's daily. The experimental results show that KDAC is advanced and effective, and can provide more generalized activation to stimulate the performance of DNER. We hope that KDAC can be exploited as a promising alternative activation function in DNER to devote itself to the construction of knowledge.


page 1

page 2

page 3

page 4


AIS: A nonlinear activation function for industrial safety engineering

In the task of Chinese named entity recognition based on deep learning, ...

Nish: A Novel Negative Stimulated Hybrid Activation Function

An activation function has a significant impact on the efficiency and ro...

Parametric Flatten-T Swish: An Adaptive Non-linear Activation Function For Deep Learning

Activation function is a key component in deep learning that performs no...

Soft-Root-Sign Activation Function

The choice of activation function in deep networks has a significant eff...

Regularization and Reparameterization Avoid Vanishing Gradients in Sigmoid-Type Networks

Deep learning requires several design choices, such as the nodes' activa...

Reborn Mechanism: Rethinking the Negative Phase Information Flow in Convolutional Neural Network

This paper proposes a novel nonlinear activation mechanism typically for...

Please sign up or login with your details

Forgot password? Click here to reset