Backpropagation through time
lastmod 2018-08-17created 2016-10-274 edits
RNNs learn their coefficients using BTPP (Backpropagation Through Time), an extension of backprop. In essence, BTPP is exactly the same as basic backprop. The only difference is that, because the structure of RNNs is connected across time, backprop is likewise applied going backward through time.
Truncated BPTT
Truncated BPTT is a simplification of BPTT, which goes back through the entire time span. As time-series data gets longer, the amount that must be stored in the hidden layer keeps growing, so storing hidden-layer values for every time step is practically impossible. Truncated BPTT is the compromise struck at a reasonable point. Because truncated BPTT does not reflect values older than the reference length, it has the problem of shortening the memory of RNNs.