Sigmoid function

lastmod 2018-08-17created 2015-12-2322 edits

A sigmoid function is a function with an S-like shape (Sigmoid curve). It is also a special form of the logistic function. It is mainly used to represent things like learning curves, and is a function that starts from small values near 0 and asymptotically approaches a certain finite value.

The equation is as follows:

\[ S(t) = \frac{1}{1 + e^{-t}} \]

The sigmoid function is also denoted by \(\sigma\) (Sigma).

Derivative

The derivation is summarized in Logistic function#Derivative. Just the result is as follows:

\[ \frac{d}{dx} \sigma(x) = \sigma(x) \cdot (1 - \sigma(x)) \]

Sigmoid neuron

• Translation: Handwriting recognition using a Neural Network

Using the sigmoid function above, a sigmoid neuron with inputs \(x_1, x_2, ...\), weights \(w_1, w_2, ...\), and bais \(b\) is as follows.

\[ \begin{eqnarray} \frac{1}{1+\exp(-\sum_j w_j x_j-b)}. \end{eqnarray} \]

This means that \(\Delta \mbox{output}\) can be expressed as a linear function of \(\Delta w_j\) and \(\Delta b\). This linearity means it is easy to change the weights and bias so that small changes in the output happen the way we want. That is why a sigmoid neuron is qualitatively very similar to a perceptron, while making it much easier to figure out how to change the weights and bias.

So how should we interpret the output of a sigmoid neuron? The biggest change between a perceptron and a sigmoid neuron is that a sigmoid neuron can have outputs other than 0 or 1. Since the output can be any real number between 0 and 1, values like 0.173 and 0.689 can be outputs. This property is very useful; for example, we can represent the average intensity of the pixels in an image as the output of a neural network. But this property can sometimes be very annoying. Suppose we have a network with two outputs: "the image is a 9" or "the image is not a 9". In this case it would obviously be much easier to express the output as 0 and 1. However, if we decide whether it is a "9" or not a "9" depending on whether the output is greater than or equal to 0.5, the problem can be easily solved.

See also

• Logistic function
• Perceptron
• Artificial neuron
• Activation function
• Artificial neural network
• Euler's number e

Favorite esite

• Wikipedia (en) description of the sigmoid function
• Sigmoid Function in an artificial neural network ¹
• Wolfram MathWorld: Sigmoid Function
• Derivative of Sigmoid
• Roboticist: Sigmoid Function
• Easy industrial mathematics - Definition of the Sigmoid activation function
• Summary of the derivative of the Sigmoid function

References