Dropout
lastmod 2018-08-17created 2015-12-292 edits
Dropout is a technique of reducing overfitting in neural networks by preventing complex co-adaptations on training data. It is a very efficient way of performing model averaging with neural networks. The term "dropout" refers to dropping out units (both hidden and visible) in a neural network.
Overview
One of the regularization techniques. During the training phase of a Deep Neural Network (not during Test), training uses only a random subset of Neurons (usually 50%) in each Hidden Layer. Doing this has many advantages.
If each training step uses only a random 50% of the Neurons, nC2 combinations arise. So it has the effect of ensembling (combining) many small neural nets within one huge DNN.
Ensembling is generally known to greatly reduce Overftting. (Because the decision is made jointly by several Classifiers rather than by a single Classifier, bias is reduced.)
Also, using Drop out reportedly reduces the number of neurons with similar Weights, and thus the neurons making redundant decisions, so each neuron can be used 100% (not actually, though).