Backpropagation

lastmod 2018-10-24created 2015-10-2618 edits

Backpropagation, also called the error backpropagation algorithm, is a statistical technique used to train multilayer perceptrons.

The multilayer perceptron is a learning structure used in machine learning. A typical multilayer perceptron consists of input layer - hidden layer - hidden layer - ... - output layer, and the layers are connected to each other by crisscrossing weight values. To train the values produced by the output layer toward the actually desired values, the statistical method of backpropagation is used.

Backpropagation is used to adjust each individual weight so that the desired value is output for the same input layer. Although it is slow, it has the advantage of producing stable results, so it is widely used in machine learning.

Backpropagation uses gradient descent.

Category

• Backpropagation:Example

Derivation

Since backpropagation uses the method, one needs to calculate the derivative of the squared error function with respect to the weights of the network. Assuming one output neuron,¹ the squared error function is:

\[ E = \tfrac{1}{2}(t - y)^2 \] , where

\[ E \] is the squared error,

\[ t \] is the target output for a training sample, and

\[ y \] is the actual output of the output neuron.

See also

• Machine learning
• Deep learning
• Gradient descent
• Convolutional neural network#CNN Backpropagation

Favorite site

• Wikipedia (en) explanation of Backpropagation
• [Recommended] A Step by Step Backpropagation Example ² (shows Backpropagation with a real example)
  • Github: Neural Network with Backpropagation (Python sample code)
• [Recommended] Backpropagation
• Coursera Neural Networks for Machine Learning Week3 - Backpropagation
• Deep learning_Neural Network_backpropagation
• [Recommended] Naver blog: Machine learning 01 ~ 18 (Korean-language machine learning blog) ³
  • [http://blog.naver.com/laonple/220513815694 Machine learning - Class 9 :<Case Study>

    Backpropagation for Neural Network]

    • Principles of training multi-layer neural network using backpropagation ⁴
• [Recommended] Lecture 3: CNN: Back-propagation ⁵
• Roboticist: Training artificial neural networks (backpropagation)
• Training a 4-layer artificial neural network (backpropagation) (derivation of the basic backpropagation formulas)
• [Recommended] Backpropagation demo (optimized for PC)
• Backpropagation - ratsgo's blog
• [Recommended] Backpropagation in CNNs - ratsgo's blog

References