Backpropagation:Example
lastmod 2018-09-14created 2015-12-2966 edits
Overview
Summarizes how to actually compute Backpropagation.
Overview
The network consists of two inputs, two hidden neurons, and two output neurons. The hidden neurons and output neurons also include a bias1.
Input / Output
<span style="color:red;">initial weights, <span style="color:orange;">the biases, <span style="color:blue;">Training inputs/outputs The values for the weights, biases, inputs, and outputs are shown in the figure on the right.
The Forward Pass
Perceptron The calculation proceeds as in a regular Feedforward neural network.
The Combination function of the first hidden neuron is as follows.
\[ net_{h1} = (w_1 * i_1) + (w_2 * i_2) + (b_1 * 1) \]
Substituting the inputs and bias into this formula gives the following.
\[ net_{h1} = (0.15 * 0.05) + (0.2 * 0.1) + (0.35 * 1) = 0.3775 \]
The combination result above is fed into the hidden neuron's Activation function. The function used is the Sigmoid function, and the formula is as follows.
\[ out_{h1} = \frac{1}{1+e^{-net_{h1}}} = \frac{1}{1+e^{-0.3775}} = 0.593269992 \]
Applying the same to the second hidden neuron (\(h_2\)) gives the following result:
\[ out_{h2} = 0.596884378 \]
Now compute the output neurons. The method is the same as above.
\[ net_{o1} = (w_5 * out_{h1}) + (w_6 * out_{h2}) + (b_2 * 1) \]
\[ net_{o1} = (0.4 * 0.593269992) + (0.45 * 0.596884378) + (0.6 * 1) = 1.105905967 \]
\[ out_{o1} = \frac{1}{1+e^{-net_{h1}}} = \frac{1}{1+e^{-1.105905967}} = 0.75136507 \]
Applying the same to the second output neuron (\(o_2\)) gives the following result:
\[ out_{o2} = 0.772928465 \]
Calculating the Total Error
Now we need to calculate the loss using a Loss function2. The loss function used is Sum of squares (Mean squared error) (MSE).
The formula is as follows.
\[ E_{output} = \sum \frac{1}{2}(target - output)^{2} \]
\(target\) is the expected output (0.01) and \(output\) is the actual output (0.75136507).
\[ E_{o1} = \frac{1}{2}(target_{o1} - out_{o1})^{2} = \frac{1}{2}(0.01 - 0.75136507)^{2} = 0.274811083 \]
Compute \(o_2\) in the same way.
\[ E_{o2} = 0.023560026 \]
Now sum all the errors to get the final error value (loss).
\[ E_{total} = E_{o1} + E_{o2} = 0.274811083 + 0.023560026 = 0.298371109 \]
The Backwards Pass
Backwards Pass
The weights are updated using backpropagation. The update uses Gradient descent.
\[ \mathbf{x}_{i+1} = \mathbf{x}_i - \gamma_i \nabla f(\mathbf{x}_i) \]
To get \(w_5^{+}\), the updated first weight \(w_5\), we need the rate of change \(\nabla f(w_5)\). So we take the Partial derivative of the loss (\(E_{total}\)) with respect to the weight (\(w_5\)) as follows. (The partial derivative is used to find the rate of change with respect to two or more unknowns.)
\[ \frac{\partial E_{total}}{\partial w_{5}} \]
However, there is no formula that allows Correlation and dependence analysis between the two variables above. (None of the formulas above has an independent variable and a dependent variable corresponding to the weight (\(w_5\)) or the loss (\(E_{total}\)).) So, to apply a formula with independent/dependent variables, let's review what we have calculated so far.
- \(E_{total} = E_{o1} + E_{o2}\)
- \(E_{o1} = \frac{1}{2}(target_{o1} - out_{o1})^{2}\)
- \(out_{o1} = \frac{1}{1+e^{-net_{o1}}}\)
- \(net_{o1} = (w_5 * h_1) + (w_6 * h_2) + (b_2 * 1)\)
The calculation in forward order (Forward Pass) is as follows.
\[ w_5 \rightarrow net_{o1} \rightarrow out_{o1} \rightarrow E_{o1} + E_{o2} \rightarrow E_{total} \]
\(E_{o2}\), which has no causal relationship, drops out during partial differentiation. Therefore \(E_{total} = E_{o1}\), and the following holds.
\[ w_5 \rightarrow net_{o1} \rightarrow out_{o1} \rightarrow E_{total} \]
Compute the above in reverse order (Backward Pass). Applying the Chain rule gives the following formula.
\[ \frac{\partial E_{total}}{\partial w_{5}} = \frac{\partial E_{total}}{\partial out_{o1}} * \frac{\partial out_{o1}}{\partial net_{o1}} * \frac{\partial net_{o1}}{\partial w_{5}} \]
Error(total) / Output(o1)
First, to find \(\frac{\partial E_{total}}{\partial out_{o1}}\), redefine \(E_{total}\).
\[ E_{o} = \frac{1}{2}(target_{o} - out_{o})^{2} \]
\[ E_{total} = E_{o1} + E_{o2} \]
\[ E_{total} = \frac{1}{2}(target_{o1} - out_{o1})^{2} + \frac{1}{2}(target_{o2} - out_{o2})^{2} \]
\[ E_{total} = \frac{1}{2}(target_{o1}^2 - 2target_{o1}out_{o1} + out_{o1}^2) + \frac{1}{2}(target_{o2}^2 - 2target_{o2}out_{o2} + out_{o2}^2) \]
\[ E_{total} = \frac{1}{2}target_{o1}^2 - target_{o1}out_{o1} + \frac{1}{2}out_{o1}^2 + \frac{1}{2}target_{o2}^2 - target_{o2}out_{o2} + \frac{1}{2}out_{o2}^2 \]
Substitute the above into the partial derivative. (Terms other than \(out_{o1}\) are treated as constants.)
\[ \frac{\partial E_{total}}{\partial out_{o1}} = -target_{o1}out_{o1}^{1 - 1} + {2}\frac{1}{2}out_{o1}^{2 - 1} \]
\[ \frac{\partial E_{total}}{\partial out_{o1}} = -target_{o1} + out_{o1} \]
\[ \frac{\partial E_{total}}{\partial out_{o1}} = -0.01 + 0.75136507 = 0.74136507 \]
Output(o1) / Network(o1)
To find \(\frac{\partial out_{o1}}{\partial net_{o1}}\), look at \(out_{o1}\).
\[ out_{o1} = \frac{1}{1+e^{-net_{o1}}} \]
\(out_{o1}\) is a Sigmoid function, and the derivative (Logistic function#Derivative) of this function has the following form.
\[ \frac{d}{dx}f(x) = f(x)\cdot(1-f(x)) \]
Here \(x\) is \(net_{o1}\) and \(f(x)\) is \(out_{o1}\), so the following holds.
\[ \frac{\partial out_{o1}}{\partial net_{o1}} = out_{o1}(1 - out_{o1}) = 0.75136507(1 - 0.75136507) = 0.186815602 \]
Network(o1) / Weight(5)
Finally, to find \(\frac{\partial net_{o1}}{\partial w_{5}}\), look at \(net_{o1}\).
\[ net_{o1} = w_5 * out_{h1} + w_6 * out_{h2} + b_2 * 1 \]
Substitute the above into the partial derivative. (Terms other than \(w_{5}\) are treated as constants.)
\[ \frac{\partial net_{o1}}{\partial w_{5}} = 1 * out_{h1} * w_5^{(1 - 1)} + 0 + 0 = out_{h1} = 0.593269992 \]
Error(total) / Weight(5)
The final result is derived as follows.
\[ \frac{\partial E_{total}}{\partial w_{5}} = \frac{\partial E_{total}}{\partial out_{o1}} * \frac{\partial out_{o1}}{\partial net_{o1}} * \frac{\partial net_{o1}}{\partial w_{5}} \]
\[ \frac{\partial E_{total}}{\partial w_{5}} = 0.74136507 * 0.186815602 * 0.593269992 = 0.082167041 \]
The modified weight(5)
To correct the error, compute the updated weight using Gradient descent. In the formula below, \(\eta\) is the learning rate. Here 0.5 is used.
\[ w_5^{+} = w_5 - \eta * \frac{\partial E_{total}}{\partial w_{5}} = 0.4 - 0.5 * 0.082167041 = 0.35891648 \]
Compute the other values in the same way.
\[ w_6^{+} = 0.408666186 \]
\[ w_7^{+} = 0.511301270 \]
\[ w_8^{+} = 0.561370121 \]
Hidden Layer
Hidden Layer
Next, we need to compute the hidden layer weights \(w_1\), \(w_2\), \(w_3\), \(w_4\).
For \(w_1\), it is as follows.
\[ \frac{\partial E_{total}}{\partial w_{1}} = \frac{\partial E_{total}}{\partial out_{h1}} * \frac{\partial out_{h1}}{\partial net_{h1}} * \frac{\partial net_{h1}}{\partial w_{1}} \]
Error(total) / Output(h1)
Note that the hidden layer affects both output layers. See the figure on the right. This can be summarized as follows.
\[ \frac{\partial E_{total}}{\partial out_{h1}} = \frac{\partial E_{o1}}{\partial out_{h1}} + \frac{\partial E_{o2}}{\partial out_{h1}} \]
\[ \frac{\partial E_{o1}}{\partial out_{h1}} = \frac{\partial E_{o1}}{\partial net_{o1}} * \frac{\partial net_{o1}}{\partial out_{h1}} \] (The step before deriving \(out_{h1}\) is \(net_{o1}\). See the #Network(o1) / Weight(5) section above.)
\(\frac{\partial E_{o1}}{\partial net_{o1}}\) can use results already calculated, as in the formula below. (\(\frac{\partial E_{o1}}{\partial out_{o1}}\) must be calculated again, but the result is the same as \(\frac{\partial E_{total}}{\partial out_{o1}}\) calculated above.)
\[ \frac{\partial E_{o1}}{\partial net_{o1}} = \frac{\partial E_{o1}}{\partial out_{o1}} * \frac{\partial out_{o1}}{\partial net_{o1}} = 0.74136507 * 0.186815602 = 0.138498562 \]
To find \(\frac{\partial net_{o1}}{\partial out_{h1}}\), look at \(net_{o1}\).
\[ net_{o1} = w_5 * out_{h1} + w_6 * out_{h2} + b_2 * 1 \]
Then take the partial derivative with respect to \(out_{h1}\). The result equals \(w_5\).
\[ \frac{\partial net_{o1}}{\partial out_{h1}} = w_5 = 0.40 \]
Substitute the results above into the formula below.
\[ \frac{\partial E_{o1}}{\partial out_{h1}} = \frac{\partial E_{o1}}{\partial net_{o1}} * \frac{\partial net_{o1}}{\partial out_{h1}} = 0.138498562 * 0.40 = 0.055399425 \]
Compute \(\frac{\partial E_{o2}}{\partial out_{o1}}\) in the same way.
\[ \frac{\partial E_{o2}}{\partial out_{h1}} = -0.019049119 \]
The final result is calculated as follows.
\[ \frac{\partial E_{total}}{\partial out_{h1}} = \frac{\partial E_{o1}}{\partial out_{h1}} + \frac{\partial E_{o2}}{\partial out_{h1}} = 0.055399425 + -0.019049119 = 0.036350306 \]
Output(h1) / Network(h1)
To find \(\frac{\partial out_{h1}}{\partial net_{h1}}\), look at \(out_{h1}\) as follows.
\[ out_{h1} = \frac{1}{1+e^{-net_{h1}}} \]
Using the derivative (Logistic function#Derivative) of the Sigmoid function, calculate as follows.
\[ \frac{\partial out_{h1}}{\partial net_{h1}} = out_{h1}(1 - out_{h1}) = 0.59326999(1 - 0.59326999 ) = 0.241300709 \]
Network(h1) / Weight(1)
To find \(\frac{\partial net_{h1}}{\partial w_1}\), look at \(net_{h1}\).
\[ net_{h1} = w_1 * i_1 + w_2 * i_2 + b_1 * 1 \]
Taking the partial derivative with respect to \(w_1\), the result equals the value of \(i_1\).
\[ \frac{\partial net_{h1}}{\partial w_1} = i_1 = 0.05 \]
Error(total) / Weight(1)
The final result is derived as follows.
\[ \frac{\partial E_{total}}{\partial w_{1}} = \frac{\partial E_{total}}{\partial out_{h1}} * \frac{\partial out_{h1}}{\partial net_{h1}} * \frac{\partial net_{h1}}{\partial w_{1}} \]
\[ \frac{\partial E_{total}}{\partial w_{1}} = 0.036350306 * 0.241300709 * 0.05 = 0.000438568 \]
The modified weight(1)
To correct the error, compute the updated weight using Gradient descent.
\[ w_1^{+} = w_1 - \eta * \frac{\partial E_{total}}{\partial w_{1}} = 0.15 - 0.5 * 0.000438568 = 0.149780716 \]
Compute \(w_2\), \(w_3\), \(w_4\) each in the same way.
\[ w_2^{+} = 0.19956143 \]
\[ w_3^{+} = 0.24975114 \]
\[ w_4^{+} = 0.29950229 \]
Python source code
# -*- coding: utf-8 -*-
import math
# INPUT1 -> WEIGHT1
# -> COMBINATION -> ACTIVATION -> OUTPUT
# INPUT2 -> WEIGHT2
def combination(w1, x1, w2, x2, bias):
return (w1 * x1) + (w2 * x2) + (1.0 * bias)
def sigmoid(x):
return 1.0 / (1.0 + math.exp(-1.0 * x))
def d_sigmoid(x):
return sigmoid(x) * (1.0 - sigmoid(x))
def cost(target, output):
return math.pow(target - output, 2.0) / 2.0
def back_error_output(target, output):
return output - target
def back_output_net(output):
return output * (1.0 - output)
def back_net_weight(input):
return input
def gradient_descent(weight, d_weight, eta=0.5):
return weight - (eta * d_weight)
class Layer(object):
def __init__(self):
self.x1 = 0.0
self.x2 = 0.0
self.w1 = 0.4
self.w2 = 0.3
self.bias = 0.5
self.threshold = 0.5
self.dw1 = 0.0
self.dw2 = 0.0
def setInput(self, x1, x2):
self.x1 = x1
self.x2 = x2
def setWeight(self, w1, w2):
self.w1 = w1
self.w2 = w2
def forward(self):
self.net = combination(self.w1, self.x1, self.w2, self.x2, self.bias)
self.output = sigmoid(self.net)
if self.output > self.threshold:
self.result = 1.0
else:
self.result = 0.0
def backward(self, target):
self.target = target
self.err = cost(self.output, target)
error_output = back_error_output(target, self.output)
output_network = back_output_net(self.output)
network_weight1 = back_net_weight(self.x1)
network_weight2 = back_net_weight(self.x2)
d_weight1 = error_output * output_network * network_weight1
d_weight2 = error_output * output_network * network_weight2
print ' - dw1: ' + str(d_weight1)
print ' - dw2: ' + str(d_weight2)
self.dw1 = gradient_descent(self.w1, d_weight1)
self.dw2 = gradient_descent(self.w2, d_weight2)
self.setWeight(self.dw1, self.dw2)
def __str__(self):
return '{}(w{}),{}(w{})={}(r{}/t{}) (e{})'.format(
self.x1, self.w1, self.x2, self.w2,
self.output, self.result, self.target, self.err)
if __name__ == '__main__':
layer = Layer()
predict = 0
while True:
layer.setInput(1, 1)
layer.forward()
layer.backward(predict)
print str(layer)
condition = raw_input('Input [q/y/n]: ')
if condition == 'q' or condition == 'Q':
break
elif condition == 'y' or condition == 'Y':
predict = 1
elif condition == 'n' or condition == 'N':
predict = 0
Backup
Backup of the current page.
Download: Backpropagation_Example_-_Open_Source_Open_Mind.pdf
See also
Favorite site
- [Recommended] A Step by Step Backpropagation Example 3 (shows Backpropagation with an actual example)
References
-
A fixed constant value. ↩
-
Or Cost function ↩
-
A_Step_by_Step_Backpropagation_Example_–_Matt_Mazur.pdf ↩