Vanishing gradient problem
lastmod 2019-05-09created 2015-12-305 edits
Gradient stability problem (Vanishing (and Exploding) Gradients)
In machine learning, the vanishing gradient problem is a difficulty found in training artificial neural networks with gradient-based learning methods and backpropagation. In such methods, each of the neural network's weights receives an update proportional to the gradient of the error function with respect to the current weight in each iteration of training. Traditional activation functions such as the hyperbolic tangent function have gradients in the range (-1, 1) or [0, 1), and backpropagation computes gradients by the chain rule. This has the effect of multiplying n of these small numbers to compute gradients of the "front" layers in an n-layer network, meaning that the gradient (error signal) decreases exponentially with n and the front layers train very slowly.
About
RNNs have a fairly long history. Several papers on RNNs had already appeared in the 1980s. Then, in the early 1990s, a problem called the vanishing gradient appeared.
The concept of a gradient is very simple. If you draw a straight line on the x-y plane, the derivative of the line by definition represents the ratio of the change in y to a small change in x. Applying this to an artificial neural network, you can find the relationship between every weight of the network and the error. In other words, you can find out how the error changes as a result of changing the network's values by a given amount. That is why computing the gradient is very important. If the gradient cannot be computed well, you cannot know the relationship between the coefficients and the error, and as a result training does not go well.
RNNs trace past hidden values by going back in time. However, the longer this tracing continues - that is, the further back into the past it goes - the more often the gradient fails to compute well.
This is because neural networks are built on multiplication operations.
With a bank savings product, multiplying even a value just slightly greater than 1 many times eventually yields a fairly large value. This is also called the magic of compound interest: even at a low interest rate, keeping money for a very long time eventually yields an enormous amount. Likewise, repeatedly multiplying a value just slightly less than 1 eventually yields a value close to 0.
The operations of an artificial neural network also consist of many multiplications, and as you keep multiplying, the gradient can vanish completely (vanishing) or diverge (exploding).
When the gradient diverges, it eventually exceeds the largest number a computer can handle. However, divergence is relatively easy to control: you just set a maximum range for the gradient. The real problem is when the gradient vanishes.
The graph below shows what happens when you multiply the sigmoid many times. After only four multiplications, the function becomes extremely flat. The same goes for backprop in RNNs. Once it becomes this flat, the slope is close to 0 over almost the entire range. In other words, the gradient does not propagate properly.
Sigmoid_vanishing_gradient.png
See also
Favorite site
- Wikipedia (en) article on the Vanishing gradient problem
- A Beginner's Guide to RNNs and LSTMs (includes the gradient stability problem)
- What is the Vanishing gradient problem?