Skip to content
Search

Stochastic gradient descent

lastmod 2019-09-26created 2016-01-1312 edits

A deep neural network can be discriminatively trained with the standard backpropagation algorithm. Here, the weights can be updated through stochastic gradient descent using the equation below.

\[ \Delta w_{ij}(t + 1) = \Delta w_{ij}(t) + \eta\frac{\partial C}{\partial w_{ij}} \]

Here, \(\eta\) is the learning rate and \(C\) is the cost function. The choice of cost function depends on factors such as the type of learning (supervised learning, unsupervised learning, reinforcement learning, etc.) and the activation function.

For example, when performing supervised learning on a multiclass classification problem, the activation function and the cost function are usually the Softmax function and the cross entropy function, respectively.

The softmax function is defined as \(p_j = \frac{\exp(x_j)}{\sum_k \exp(x_k)}\), where \(p_j\) is the class probability, and \(x_j\) and \(x_k\) are the total input to unit \(j\) and the total input to unit \(k\), respectively.

Cross entropy is defined as \(C = -\sum_j d_j \log(p_j)\), where \(d_j\) is the target probability for output unit \(j\), and \(p_j\) is the probability output for \(j\) after applying the activation function.

Comparison of a few optimization methods

Comparison of a few optimization methods (animation by Alec Radford). The star denotes the global minimum on the error surface. Notice that stochastic gradient descent (SGD) without momentum is the slowest method to converge in this example. We're using Nesterov's Accelerated Gradient Descent (NAG) throughout this tutorial.

Machine_learning_-_Comparison_of_a_few_optimization_methods.gif

Momentum

If the objective has the form of a long shallow ravine leading to the optimum and steep walls on the sides, standard SGD will tend to oscillate across the narrow ravine since the negative gradient will point down one of the steep sides rather than along the ravine towards the optimum.

The objectives of deep architectures have this form near local optima and thus standard SGD can lead to very slow convergence particularly after the initial steep gains.

Momentum is one method for pushing the objective more quickly along the shallow ravine. (Momentum is a method for driving toward the objective faster along a shallow ravine.) 

v=γv+α∇θJ(θ;x(i),y(i))
θ=θ−v

In the above equation v is the current velocity vector which is of the same dimension as the parameter vector θ. The learning rate α is as described above, although when using momentum α may need to be smaller since the magnitude of the gradient will be larger. Finally γ∈(0,1] determines for how many iterations the previous gradients are incorporated into the current update. Generally γ is set to 0.5 until the initial learning stabilizes and then is increased to 0.9 or higher.

See also

Favorite site

References

  1. Daniel_Nouris_Blog_-_Using_convolutional_neural_nets_to_detect_facial_keypoints_tutorial.pdf ↩

sitemap