Batch Normalization

lastmod 2022-03-21created 2016-10-269 edits

Batch normalization is an artificial neural network technique that keeps the inputs to a layer from becoming skewed to one side, too spread out, or too narrow. For a batch made up of multiple inputs, it re-adjusts the mean and standard deviation of each layer's inputs, preventing the inputs from becoming skewed.

Batch Normalization is basically one of the ideas for preventing Gradient Vanishing / Gradient Exploding. Until then, this problem had been addressed by changing the Activation function (ReLU, etc.), Careful Initialization, small learning rate, and so on, but this paper wanted to find a fundamental way to accelerate training by stabilizing the training process itself as a whole, rather than through such indirect methods.

They argue that the reason this instability occurs is ‘Internal Covariance Shift’. The phenomenon called Internal Covariance Shift refers to the distribution of the input changing at each layer or Activation of the Network. To prevent this, one can simply think of normalizing the input distribution of each layer to an input with mean 0 and standard deviation 1, which can be done with a method called whitening. Whitening is basically an operation that makes the features of the incoming input uncorrelated and makes the variance of each one 1.

The problem is that whitening requires computing the covariance matrix and its inverse, so not only is it computationally expensive, but to make matters worse, whitening causes the effect of some parameters to be ignored. For example, consider a network that takes input u, outputs x=u+b, and tries to learn an appropriate bias b, and suppose we subtract E[x] from x. In that case, the value of b is subtracted along with E[x], and in the end the effect of b disappears from the output. If the process goes beyond simply subtracting E[x] and includes scaling, such as dividing by the standard deviation, this tendency gets even worse, and the paper reportedly confirmed this experimentally.

To make up for these drawbacks of whitening while reducing internal covariance shift, the paper took the following approach.

• Assume that the features are already uncorrelated, compute the mean and variance in scalar form for each feature only, and normalize each one.
• Simply fixing the mean and variance to 0 and 1 can actually remove the nonlinearity of the Activation function. For example, if the input to a sigmoid activation has mean 0 and variance 1, the output will be closer to a straight line than a curve. Also, the assumption that the features are uncorrelated can limit what the network can represent. To make up for these points, a scale factor (gamma) and a shift factor (beta) are added to the normalized values, and these variables are trained together during back-prop.
• Instead of computing the mean and variance over the entire training data, it computes them per mini-batch. The mean and variance are computed only within the currently selected mini-batch, and these values are used to normalize.

Documentation

Batch Normalization - Accelerating Deep Network Training by Reducing Internal Covariate Shift

https://arxiv.org/abs/1502.03167

See also

• Deep Learning
• Gradient Vanishing
• Gradient Exploding

Favorite site

• Batch Normalization explanation and implementation ¹
• Batch Normalization (ICML 2015)
• Artificial neural networks ( ANN ) #6-2 Optimization: initial weight setup, gradient vanishing ( gradient vanishing ), batch normalization ( batch normalization ) :: UMBUM
• Batch Normalization explanation and implementation
• Uses of Batch Normalization (study notes)
• Deep learning even liberal arts students can understand (10) - Batch normalization

References