{Contrastive Instance Discrimination}
The key concept of contrastive learning is to generate the positive and negative training sample pairs based on the understanding of the data. The model needs to learn a function so that the two positive samples have high similarity scores and two negative samples have low similarity scores. As a result, the appropriate sample generation is essential to ensure the model learn the underlying features/structures of the data.
ImageNet
MS-COCO
DABS (Domain-Agnostic Benchmark for Self-supervised learning) [Paper]
STL-10
A fundamental idea behind most machine learning algorithms is that similar examples should be grouped together and far from other clusters of related examples.
It pulls representations of augmented versions/views of the same image closer to each other (contracting positives), while simultaneously pushing different images away from each other (contrasting negatives) in representation space.
Image Field
The contrastive learning in the image field applies two different data augmentation from the same original image to generate positive sample pairs and use two different images as the negative sample pairs.
Challenges:
The strength of augmentation. If the augmentation is so strong that there is no relationship between the two augmented samples from the same sample, the model cannot learn. Likewise, if the augmentation is so little that the model can easily solve the problem, then the model also cannot learn useful information for downstream tasks.
The selection of negative sample pairs. If we randomly assign two images as a negative pair, they can be the same class (e.g., two images of cats), which introduces conflicting noise to the model. If the negative pairs are very easy to distinguish, then the model cannot learn the underlying features/structures of data.
Speech Field
One approach is applying augmentation like SimCLR (Speech SimCLR).
Another approach is to use adjacent features as the positive pairs and features from different samples as the negative pairs (e.g, CPC, Wav2vec (v1, v2.0), VQ-wav2vec and Discret BERT).
Graph Field
DGI maximized the mutual information between patch representations and global representations of the graphs, and minimized the mutual information between the patch representation of corrupted graphs and the global representation of original graph.
Text Field
Classification maximizes the output of positive class and minimizes the outputs of negative classes. Likewise, contrastive learning also maximizes the output of positive pairs and minimizes the outputs of negative pairs.
The key distinction is that classification has finite amount of negative classes (in case of text tokens) whereas contrastive learning has infinite amount of negative classes (in case of images and acoustic features).
Contrastive learning achieves this by using three key ingredients:
Positive (s)
Anchor
Negative(s).
Consider a pair (x, y). Conceptually, contrastive embedding methods take a convolutional network, and feed x and y through this network to obtain two feature vectors: h and h’.
Contrastive Learning and Loss Function Abstraction
To create a positive pair (x, y), we need two examples that are similar.
x and y have the same content, such that y is a transformation of x that preserves its content (rotation, magnification, cropping, etc.).
Fig. 1: Positive Pair
For a negative pair, we use a third sample that is not similar.
(x_neg, y_neg): Negative samples with different content/ class label.
Fig. 2: Negative Pair
We push down on the energy (maximize the similarity) of similar pairs, while pushing up on the energy (minimize the similarity) of dissimilar pairs.
Produce features that are good for object recognition that can rival the features learned through supervised methods.
InfoNCE (van den Oord et al. 2018) is a lower bound to MI between views:
MI = I(v1; v2) ≥ I(z1; z2) ≥ log(K) − ℒInfoNCE
Minimizing InfoNCE leads to maximizing the MI between view v1 and view v2.
Optimal views are at the sweet spot where it only encodes useful information for transfer:
Minimal sufficient encoder depends on downstream tasks (Tian et al. 2020)
Composite loss for finding the sweet spot (Tsai et al. 2020)
Contrastively learned features are more uniform and aligned.
Uniform: features should be distributed uniformly on the hypersphere Sd
Aligned: features from two views of the same input should be the same.
Contrastive methods sometimes suffer from dimensional collapse (Hua et al. 2021)
Features span lower-dimensional subspace instead
Two causes demonstrated by Jing et al. (2021)
Strong augmentation & implicit regularization
Sampling complexity decreases when:
Adopting contrastive learning objectives (Arora et al. 2019)
Predicting the known distribution in the data (Lee et al. 2020)
Linear classifier on learned representation is nearly optimal (Tosh et al. 2021)
Spectral Contrastive Learning (HaoChen et al. 2021)
Instance Discrimination applies the concept of contrastive learning to whole image instance.
Instance Discrimination method constraints that two augmented versions of the same image (positive pair) should have similar representations and two augmented versions of the different image (negative pair) should have different representations.
SimCLR: a simple contrastive learning framework that trains a model to maximize the agreement between different views of the same instance.
MoCo: a method that uses a dynamically updated memory bank to store negative examples for contrastive learning.
SwAV: a method that combines the strengths of SimCLR and MoCo for large-scale visual representation learning.
BYOL: a method that uses an online network to generate representations for a target network to contrast against.
InfoNCE: a contrastive learning method that maximizes the mutual information between the representations of positive and negative examples.
CPC: a contrastive learning method that trains a model to predict the context in which an instance will be used in the future.
PIRL: a method that uses positive instance-level relationships to learn representations in a contrastive manner.
The main objective is, under a certain kind of image augmentations, the learnt representations should be invariant. These certain image augmentations include horizontal flip, a random crop of a certain size, color channel distortion, gaussian blur, etc.
Intuitively, these augmentations although change the input image, but does not change the class of the input image (a cat would be a cat after flipping and cropping as well) and hence their representations should also not change.
Steps:
Given an image, create an image pair <x1, x2> with two randomly augmented versions of this image. Sample N-2 negative samples by randomly taking an augmented version of any other image in the dataset.
Pass this image pair to the encoder model (g_enc), and get the representations. Also, obtain the representations of the N-2 negatives.
Similar to CPC, apply InfoNCE loss to induce similarity between positive pairs and dissimilarity between negative pairs.
The major difference between SimCLR and MoCo is how they handle the negative samples.
Objective function: NCE (Noise Contrastive Estimator).
Here we define the similarity metric between two feature maps/vectors as the cosine similarity.
What PIRL does differently is that it doesn’t use the direct output of the convolutional feature extractor. It instead defines different heads ff and gg, which can be thought of as independent layers on top of the base convolutional feature extractor.
PIRL’s NCE objective function works:.
In a mini-batch, we will have one positive (similar) pair and many negative (dissimilar) pairs.
We then compute the similarity between the transformed image’s feature vector (I^t) and the rest of the feature vectors in the minibatch (one positive, the rest negative).
We then compute the score of a softmax-like function on the positive pair. Maximizing a softmax score means minimizing the rest of the scores, which is exactly what we want for an energy-based model.
The final loss function, therefore, allows us to build a model that pushes the energy down on similar pairs while pushing it up on dissimilar pairs.
Dr. LeCun mentions that to make this work, it requires a large number of negative samples. In SGD, it can be difficult to consistently maintain a large number of these negative samples from mini-batches. Therefore, PIRL also uses a cached memory bank.
Question: Why do we use cosine similarity instead of L2 Norm?
With an L2 norm, it’s very easy to make two vectors similar by making them “short” (close to centre) or make two vectors dissimilar by making them very “long” (away from the centre).
This is because the L2 norm is just a sum of squared partial differences between the vectors. Thus, using cosine similarity forces the system to find a good solution without “cheating” by making vectors short or long.
Fig. PIRL's Results on ImageNet
PIRL is starting to approach the top-1 linear accuracy of supervised baselines (~75%).
The technique uses a sophisticated data augmentation method to generate similar pairs, and they train for a massive amount of time (with very, very large batch sizes) on TPUs.
Dr. LeCun believes that SimCLR, to a certain extent, shows the limit of contrastive methods.
There are many, many regions in a high-dimensional space where you need to push up the energy to make sure it’s actually higher than on the data manifold.
As you increase the dimension of the representation, you need more and more negative samples to make sure the energy is higher in those places not on the manifold.
Fig. SimCLR's Results on ImageNet
Contrastive divergence.
Persistent Contrastive Divergence
Ratio Matching.
Noise Contrastive Estimation.
Minimum Probability Flow.
Same Class => Similar Embeddings
Different Class => Dissimilar Embeddings
n-pair loss: https://papers.nips.cc/paper/6200-improved-deep-metric-learning-with-multi-class-n-pair-loss-objective
ranking-based NCE: https://arxiv.org/abs/1809.01812
Poole et al., 2019: https://arxiv.org/abs/1905.06922
Characteristic 1: Data Augmentation Pipeline.
Characteristic 2: Encoder.
Characteristic 3: Representation Extraction.
Characteristic 4: Similarity Measure.
Characteristic 5: Loss Function.
