Normalization

In a deep neural network, there is a phenomenon called internal covariate shift, which is a change in the input distribution to the network's layers due to the ever-changing network parameters during training.

The input layer may have certain features which dominate the process, due to having high numerical values:

- For example, imagine feature one having values between 1 and 5, and feature two having values between 100 and 10000.
- During training, due to the difference in scale of both features, feature two would dominate the network and only that feature would have a contribution to the outcome of the model.
- This can create a bias in the network because only those features contribute to the outcome of the training.

Due to the reasons stated, a concept known as normalization was introduced to resolve these issues.

0) Contents:

Batch Normalization
Layer Normalization
Instace Normalization
Group Normalization
Weight Standardization

N,C,H,W referring to the batches, channels, height, and width respectively.
Batch normalization normalizes in the N direction.
Layer normalization and group normalization normalizes in the C direction but group normalization as the name suggests divides the channels(C) into groups and normalizes them separately.

1) Batch Normalization:

Batch normalization enables a larger learning rate and faster convergence by reducing the internal covariate shift during training CNNs.
Scales the inputs to a layer to a common value for every mini-batch during the training of deep neural networks.
Estabilizes the learning process and significantly reduces the number of epochs required to train deep networks, enabling the network to train faster.

The way batch normalization works are by calculating the mean and variance of every feature in the mini-batch, then the mean is subtracted and each feature is divided by the standard deviation of the mini-batch.

2) Layer Normalization:

Normalizes the summed input across the features as opposed to normalizing the input features across the mini-batches, as in the case of batch normalization.
Layer normalization can be applied to recurrent neural networks by normalization separately at each time step.

i refers to the batches and j refers to the features, so as can be seen from the equation, the features are also normalized and not just the batches.

3) Instance Normalization:

Instance normalization uses the statistics of an individual sample instead of a mini-batch to normalize features.
Normalizes across each channel of the training data.
Instance normalization can also be used on the test data because it doesn't depend on the mini-batch.
Instance normalization plays a big role in the style transfer of the generative adversarial network. Instance normalization is sometimes called contrast normalization.

k and j represent the image height and width, "i" represent the channel(if the input is RGB, then it is a color channel), the index of the image in the batch is represented by t, all denoted by the stick.

Notes: Use instance normalization for image classification where the class label should not depend on the contrast of the input image.

Color distribution still plays a role. It is more likely to be an apple than an orange if it has a lot of red.
At later layers, you can no longer imagine instance normalization acts as contrast normalization. Class-specific details will emerge in deeper layers and normalizing them by instance will hurt the model's performance greatly.
IBN-Net uses both batch normalization and instance normalization in their model. They only put instance normalization in early layers and have achieved improvement in both accuracy and ability to generalize.

IN provide visual and appearance in-variance and BN accelerate training and preserve discriminative feature. IN is preferred in Shallow layer(starting layer of CNN) so remove appearance variation and BN is preferred in deep layers(last CNN layer) should be reduce in order to maintain discrimination.

3) Group Normalization:

Divides the channels into groups and normalizes them for each training example.
This normalization can be said to be a hybrid, can be represented as:
1. Instance normalization by putting each channel into different groups.
2. Layer normalization by putting all channels into a single group.
Doesn't depend on batch sizes.
Group Normalization computes the mean and variance along the height and width axes and along with a group of a number of channels. Group normalization has the capacity to replace batch normalization for a number of tasks.

x is the feature and "i" is the index.
N is the batch, C is the channel, H and W as usual are the height and width. G refers to the number of groups, with C/g indicating the number of channels in a group.

References:

Page updated

Google Sites

Report abuse

Normalization

About Me: