Activation function
lastmod 2026-02-26created 2015-12-2218 edits
ArtificialNeuronModel_english.png
In an artificial neural network model, the main function of a neuron is to compute the weighted sum NET of its inputs and connection strengths and then emit an output through the activation function. Therefore, the neuron's output may differ depending on which activation function is chosen.
An activation function must be a monotonically increasing function and can generally be classified as follows. Because these classification methods are interrelated, rather than strictly separating them, this page describes mainly the commonly used functions to aid understanding.
Type
- Unipolar / bipolar functions
- Linear / nonlinear functions
- Continuous / binary functions
Sigmoid function
The sigmoid function is a unipolar or bipolar nonlinear continuous function, as shown in the figure below, and is the most widely used activation function in neural network models. Because its shape is S-shaped, the sigmoid function is also called an S-shaped curve. See the corresponding page for details.
Logistic-curve.svg.png
Effective Activation function
- ReLU / Rectified-Linear and Leaky-ReLU
- Sigmoid function
- TanH / Hyperbolic Tangent
- Absolute Value
- Power
- BNLL
Comparison of activation functions
Some desirable properties in an activation function include:
- Nonlinear: When the activation function is non-linear, then a two-layer neural network can be proven to be a universal function approximator. The identity activation function does not satisfy this property. When multiple layers use the identity activation function, the entire network is equivalent to a single-layer model.
- Continuously differentiable: This property is necessary for enabling gradient-based optimization methods. The binary step activation function is not differentiable at 0, and it differentiates to 0 for all other values, so gradient-based methods can make no progress with it.
- Monotonic: When the activation function is monotonic, the error surface associated with a single-layer model is guaranteed to be convex.
- \(f(x)\approx x\) when \(x \approx 0\): This property enables the neural network to train efficiently when its weights are initialized with small random values. When the activation function does not satisfy this property, special care must be used when initializing the weights.
- Range: When the range of the activation function is finite, gradient-based training methods tend to be more stable, because pattern presentations significantly affect only limited weights. When the range is infinite, training is generally more efficient because pattern presentations significantly affect most of the weights. In the latter case, smaller learning rates are typically necessary.
The following table compares the properties of several activation functions:
| Name | Plot | Monotonic | \(f(x)\approx x\) when \(x \approx 0\) | Range | ||
| Identity | | \(f(x)=x\) | \(f'(x)=1\) | Yes | Yes | \((-\infty,\infty)\) |
| Binary step | | \(f(x) = \left \{ \begin{array}{rcl} 0 & \mbox{for} & x < 0\\ 1 & \mbox{for} & x \ge 0\end{array} \right.\) | \(f'(x) = \left \{ \begin{array}{rcl} 0 & \mbox{for} & x \ne 0\\ ? & \mbox{for} & x = 0\end{array} \right.\) | Yes | No | \(\{0,1\}\) |
| Logistic (a.k.a Soft step) | | \(f(x)=\frac{1}{1+e^{-x}}\) | \(f'(x)=f(x)(1-f(x))\) | Yes | No | \((0,1)\) |
| TanH | | \(f(x)=\tanh(x)=\frac{2}{1+e^{-2x}}-1\) | \(f'(x)=1-f(x)^2\) | Yes | Yes | \((-1,1)\) |
| ArcTan | | \(f(x)=\tan^{-1}(x)\) | \(f'(x)=\frac{1}{x^2+1}\) | Yes | Yes | \((-\frac{\pi}{2},\frac{\pi}{2})\) |
| | \(f(x) = \left \{ \begin{array}{rcl} 0 & \mbox{for} & x < 0\\ x & \mbox{for} & x \ge 0\end{array} \right.\) | \(f'(x) = \left \{ \begin{array}{rcl} 0 & \mbox{for} & x < 0\\ 1 & \mbox{for} & x \ge 0\end{array} \right.\) | Yes | No | \([0,\infty)\) | |
| SoftPlus | | \(f(x)=\log_e(1+e^x)\) | \(f'(x)=\frac{1}{1+e^{-x}}\) | Yes | No | \((0,\infty)\) |
| Bent identity | | \(f(x)=\frac{\sqrt{x^2 + 1} - 1}{2} + x\) | \(f'(x)=\frac{x}{2\sqrt{x^2 + 1}} + 1\) | Yes | Yes | \((-\infty,\infty)\) |
| SoftExponential | | \(f(\alpha,x) = \left \{ \begin{array}{rcl} -\frac{\log_e(1-\alpha (x + \alpha))}{\alpha} & \mbox{for} & \alpha < 0\\ x & \mbox{for} & \alpha = 0\\ \frac{e^{\alpha x} - 1}{\alpha} + \alpha & \mbox{for} & \alpha > 0\end{array}\right.\) | \(f'(\alpha,x) = \left \{ \begin{array}{rcl} \frac{1}{1-\alpha (\alpha + x)} & \mbox{for} & \alpha < 0\\ e^{\alpha x} & \mbox{for} & \alpha \ge 0\end{array} \right.\) | Yes | Yes iff \(\alpha\approx0\) | \((-\infty,\infty)\) |
| Sinusoid | | \(f(x)=\sin(x)\) | \(f'(x)=\cos(x)\) | No | Yes | \([-1,1]\) |
| Sinc | | \(f(x)=\left \{ \begin{array}{rcl} 1 & \mbox{for} & x = 0\\ \frac{\sin(x)}{x} & \mbox{for} & x \ne 0\end{array} \right.\) | \(f'(x)=\left \{ \begin{array}{rcl} 0 & \mbox{for} & x = 0\\ \frac{\cos(x)}{x} - \frac{\sin(x)}{x^2} & \mbox{for} & x \ne 0\end{array} \right.\) | No | No | \([\approx-.217234,1]\) |
| Gaussian | | \(f(x)=e^{-x^2}\) | \(f'(x)=-2xe^{-x^2}\) | No | No | \((0,1]\) |