What is an Activation Function?
An activation function determines the output behavior of each node, or “neuron” in an artificial neural network.
Activation functions are crucial basic components of artificial neural networks (ANN), since they introduce non-linear characteristics to the network. This allows the ANN to learn complicated, non-linear mappings between inputs and response variables.
Without these non-linear activation functions, then nodal activation would be limited to a linear combination of the inputs, which would exponentially increase the processing power and time needed to solve problems and severely limit the types of relationships between data set features that could be discovered.
How do Activation Functions Work?
While there are countless variations of these functions, most network frameworks begin by computing the weighted sum of the inputs. Each node in the layer can have it's one unique weighting. However the activation is function is the same across all nodes in the layer. While weights are learning parameters, the activation functions are typically of a fixed form.