Contractive Autoencoders

Contractive autoencoders perform task of learning a representation of the image while passing it through a bottleneck and reconstructing it in the decoder.

The contractive autoencoder also has a regularization term to prevent the network from learning the identity function and mapping input into the output.

Contractive autoencoders work on the basis that similar inputs should have similar encodings and a similar latent space representation. It means that the latent space should not vary by a huge amount for minor variations in the input.

To train a model that works along with this constraint, we have to ensure that the "derivatives of the hidden layer activations" are small with respect to the input.

"Denoising autoencoders make the reconstruction function (ie. decoder) resist small but ﬁnite-sized perturbations of the input, while contractive autoencoders make the feature extraction function (ie. encoder) resist infinitesimal perturbations of the input."

Model

Source: https://www.v7labs.com/blog/autoencoders-guide

Loss Function

Reconstruction Loss

- Wants the model to tell differences between two inputs and observe variations in the data.

Derivatives Loss

- The Frobenius norm of the derivatives says that the model should be able to ignore variations in the input data.

Total Loss

- Putting these two contradictory conditions into one loss function enables us to train a network where the hidden layers now capture only the most essential information.
- This information is necessary to separate images and ignore information that is non-discriminatory in nature, and therefore, not important.

References

https://www.v7labs.com/blog/autoencoders-guide

Page updated

Google Sites

Report abuse