Rectifier
lastmod 2018-08-17created 2015-12-244 edits
Activation_rectified_linear.svg.png
In the context of artificial neural networks, the rectifier is an activation function defined as
\[ f(x) = \max(0, x) \]
Overview
One of the activation functions. It is powerful because gradient descent requires differentiating composite functions, which means multiplying by the derivative of the activation function, and the existing sigmoid has a maximum slope of 1/4, so each time gradient descent proceeds through another layer, the error vanishes by \((1/4)^2\).
With ReLU, however, the slope is 0 or 1, so whenever learning occurs, 100% of the error is propagated and this problem does not arise. Also, since the activation value is not bounded to [0,1] as with sigmoid but unbounded, it can be seen as having stronger expressive power. As a result, both regularization and representation become more powerful.