Self-Supervised Learning

{Self-Prediction; Contrastive Learning}

Self-supervised representation learning aims to obtain robust representations of samples from raw data without expensive labels or annotations. Early methods in this field focused on defining pre-training tasks which involved a surrogate task on a domain with ample weak supervision labels.

Encoders trained to solve such tasks are expected to learn general features that might be useful for other downstream tasks requiring expensive annotations like image classification.

Motivation, Objectives and Related Works

Motivation

Motivated by the idea of constructing supervised learning tasks out of unsupervised datasets.
1. Data labeling is expensive and thus high-quality labeled dataset is limited.
2. Learning good representation makes it easier to transfer useful information to a variety of downstream tasks.
  - A downstream task has only a few examples.
  - Zero-shot transfer to new tasks.

Objectives

Self-Supervised Learning (SSL) is a special type of representation learning that enables learning good data representation from an unlabelled dataset.

Basis supervised learning definition: The network follows supervised learning where labels are obtained in a semi-automated manner, without human input.
Prediction problem: a part of the data is hidden, and rest visible. Hence, the aim is to either predict the hidden data or to predict some property of the hidden data.

Computer vision pipelines (employing SSL) = a pretext task + a real (downstream) task

The pretext task: is solved to learn visual representations, with the aim of using the learned representations or model weights obtained in the process, for the downstream task.
The downstream task: (knowledge transfer process) can be anything like classification or detection task, with insufficient annotated data samples.

Self-supervised learning Vs Unsupervised Learning

Self-supervised learning and unsupervised learning methods can be considered complementary learning techniques as both do not need labeled datasets.
- Unsupervised learning does not have any feedback loops while self-supervised learning has a lot of supervisory signals that act as feedback in the training process.
- Unsupervised learning methods are good at clustering, and dimensionality reduction, while self-supervised learning is a pretext method for regression and classification tasks.

How self-supervised learning works

The task is solved based on pseudo-labels which help to initialize the network weights.
The actual task is performed with supervised or unsupervised learning.

Related Works

Restricted Boltzmann Machines
- Harmonium (Smolensky 1986)
- Contrastive divergence (Hinton 2000; Hinton 2002)
- Greedy layer-wise pre-training (Hinton et al. 2006; Bengio et al. 2007)
Autoencoder:
- Denoising Autoencoder (Vincent et al. 2008)
Word2Vec: Self-Supervised Learning for Language
- Given an input sentence, the task involves predicting a missing word from that sentence, which is specifically omitted for the purpose of building a pretext task.
- Hence, the set of labels becomes all possible words in the vocabulary, and, the correct label is the word that was omitted from the sentence.
- Thus, the network can then be trained using regular gradient-based methods to learn word-level representations.
- Word embeddings to map words to vectors.
- CBOW & Skip-gram (Mikolov et al. 2013): Neighboring words → middle word (CBOW, and Word → neighboring words (skip-gram)
- GloVe (Pennington et al. 2014): Log-bilinear on word co-occurrences.
Autoregressive Modeling:
- Hidden Markov Models (Baum & Petrie 1966)
- Recurrent Neural Networks (Williams, Hinton, & Rumelhart 1986)
- Neural Autoregressive Distribution Estimator (Larochelle et al. 2011)
Siamese Networks:
- Self-organizing Neural Networks (Becker & Hinton 1992)
- Siamese networks (Bromley et al. 1994)
Multiple Instance Learning & Metric Learning:
- Multidimensional scaling (MDS; Cox et al. 1994)
- Locally linear embedding (LLE; Roweis et al. 2000)
- Metric learning (Xing et al. 2002)
- Contrastive Loss (Chopra & Hadsell et al. 2005)
- Triplet loss (Schroff et al. 2015)
- N-pair loss (Sohn 2016)

Figure. Diagrams for SimCLR, MoCo, BYOL, SimSiam and VICReg

Methodology

Contemporary self-supervised learning methods can roughly be broken down into two classes of methods [Link]

Generative

Given an individual data sample, the task is to predict one part of the sample given the other part.
The part to be predicted pretends to be missing.

Self-prediction construct prediction tasks within every individual data sample: to predict a part of the data from the rest while pretending we don’t know that part.

Autoregressive Generation

- The autoregressive model predicts future behavior based on past behavior.
- Any data that comes with an innate sequential order can be modeled with regression.
- Examples:
  1. Audio (WaveNet, WaveRNN);
  2. Autoregressive language modeling (GPT, XLNet);
  3. Images in raster scan (PixelCNN, PixelRNN, iGPT)

Masked Generation

- We mask a random portion of information and pretend it is missing, irrespective of the natural sequence.
- The model learns to predict the missing portion given other unmasked information.
- Examples:
  1. Masked language modeling (BERT)
  2. Images with masked patch (denoising autoencoder, context autoencoder, colorization)

Innate Relationship Prediction

- Some transformation (e.g. segmentation, rotation) of one data sample should maintain the original information or follow the desired innate logic.
- Examples:
  1. Order of image patches (e.g., relative position, jigsaw puzzle).
  2. Image rotation.
  3. Counting features across patches.

Hybrid Self-Prediction Models

1. VQ-VAE + AR
  - Jukebox (Dhariwal et al. 2020)
  - DALL-E (Ramesh et al. 2021)
2. VQ-VAE + AR + Adversarial
  - VQGAN (Esser & Rombach et al.2021)

Contrastive

Given multiple data samples, the task is to predict the relationship among them.
The multiple samples can be selected from the dataset based on some known logics (e.g. the order of words/ sentences), or fabricated by altering the original version.

The goal of contrastive representation learning is to learn such an embedding space in which: Similar sample pairs stay close to each other + Dissimilar ones are far apart.

Inter-sample Classification

Given both similar (“positive”) and dissimilar (“negative”) candidates, to identify which ones are similar to the anchor data point is a classification task.
There are creative ways to construct a set of data point candidates:
- The original input and its distorted version.
- Data that captures the same target from different views.
Common loss functions:

Contrastive loss (Chopra et al. 2005)
Triplet loss (Schroff et al. 2015; FaceNet)
Lifted structured loss (Song et al. 2015)
Multi-class N-pair loss (Sohn 2016)
Noise contrastive estimation (“NCE”; Gutmann & Hyvarinen 2010)
InfoNCE (van den Oord, et al. 2018)
Soft-nearest neighbors loss (Salakhutdinov & Hinton 2007, Frosst et al. 2019)

1. Contrastive loss (Chopra et al. 2005):

Learning a Similarity Metric Discriminatively, with Application to Face Verification.
Works with labelled dataset.
Encodes data into an embedding vector such that examples from the same class have similar embeddings and samples from different classes have different ones.
Given two labeled data pairs (xi, yi) and (xj, yj):

The contrastive loss encourages the model to minimize the loss for similar pairs (making their distances small) and maximize the loss for dissimilar pairs (making their distances larger than the margin).

This way, it learns to create a clear separation in the embedding space between similar and dissimilar pair

import torch

import torch.nn as nn

import torch.nn.functional as F

class ContrastiveLoss(nn.Module):

def __init__(self, margin = 1.0):

super(ContrastiveLoss, self).__init__()

self.margin = margin

def forward(self, output1, output2, label):

# calculate Euclidean distance

euclidean_distance = F.pairwise_distance(output1, output2)

# calculate loss_contrastive

loss_contrastive = torch.mean((1-label) * torch.pow(euclidean_distance, 2) +

(label) * torch.pow(torch.clamp(self.margin - euclidean_distance, min=0.0), 2))

return loss_contrastive

2. Triplet loss (Schroff et al. 2015):

FaceNet: A Unified Embedding for Face Recognition and Clustering.
Learns to minimize the distance between the anchor x and positive x+; and maximize the distance between the anchor x and negative x- at the same time.
L(A,P,N) = max{d(A,P) − d(A,N) + α, 0}
Given a triplet input (x, x+, x-):

It encourages the network to learn embeddings (feature representations) in such a way that similar data points are closer together in the embedding space while dissimilar data points are farther apart.

class TripletLoss(nn.Module):

def __init__(self, margin = 1.0):

super(TripletLoss, self).__init__()

self.margin = margin

def forward(self, anchor, positive, negative):

# Compute the Euclidean distances

distance_positive = torch.norm(anchor - positive, 2, dim=1) # dim=1: row-wise

distance_negative = torch.norm(anchor - negative, 2, dim=1)

# Compute the loss

loss_triplet = torch.mean(torch.clamp(d_p - d_n + self.margin, min=0.0))

return loss_triplet

3. N-pair loss (Sohn 2016):

Improved Deep Metric Learning with Multi-class N-pair Loss Objective.
Generalizes triplet loss to include comparison with multiple negative samples.
It's designed to train deep neural networks to learn embeddings (feature representations) in a way that enhances the discrimination between classes or categories in the embedding space.
Given one positive and N-1 negative samples {x, x+, x-1, ..., x-N-1},

Deep metric learning with triplet loss (left) and (N+1)-tuplet loss (right).

The N-pair loss is used for multi-class classification or similarity learning tasks, where each sample in the batch belongs to one of multiple classes or categories.

By taking the negative log-softmax, the network is trained to maximize the similarity between a sample and its correct class while minimizing the similarity with other classes. This promotes better class separation in the embedding space

Efficient batch construction via N-pair examples: O(N^2)→O(N)
- N tuples of (N+1)-tuplet loss requires N(N+!) examples to be evaluated.
- We can obtain N tuples of (N+1)-tuplet loss by constructing a batch with N pairs whose pairs are from different classes. This requires only 2N examples to be evaluated.

Multi-class N-pair (N-pair-mc) loss:

4. Lifted Structured Loss (Song et al. 2015):

Utilizes all the pairwise edges within one training batch for better computational efficiency.

Illustration for a training batch with six examples. Red edges and blue edges represent similar and dissimilar examples respectively. In contrast, Song's method explicitly takes into account all pair wise edges within the batch.

5. Noise Contrastive Estimation (NCE) (Gutmann & Hyvarinen 2010):

Runs logistic regression to tell apart the target data from noise.
Given target sample distribution p and noise distribution q,

6. InfoNCE (van den Oord, et al. 2018):

Uses categorical cross-entropy loss to identify the positive sample amongst a set of unrelated noise samples.
Given a context vector c, the positive sample should be drawn from the conditional distribution p(x|c), while N−1 negative samples are drawn from the proposal distribution p(x), independent from the context c.

7. Soft-Nearest Neighbors Loss (Frosst et al. 2019):

Extends the loss function to include multiple positive samples given known labels.

Feature Clustering

Find similar data samples by clustering them with learned features.
Core idea: Use clustering algorithms to assign pseudo labels to samples such that we can run intra-sample contrastive learning.
- Feature Learning: Train a neural network to learn meaningful representations (features or embeddings) from the raw data. These representations should capture the underlying structure of the data.
- Clustering: Apply a clustering algorithm (e.g., K-means, hierarchical clustering, spectral clustering) to group the data samples based on their learned representations. Each cluster represents a group of similar samples.
- Pseudo-Labeling: Assign pseudo-labels to the samples within each cluster. These pseudo-labels are based on the cluster assignments and represent the cluster membership of each sample.
- Intra-Sample Contrastive Learning: Use these pseudo-labels to create contrastive pairs within each cluster for training a contrastive loss. The goal is to encourage the model to pull similar samples closer in the embedding space and push dissimilar samples further apart within each cluster.
Examples:
- DeepCluster (Caron et al 2018);
- InterCLR (Xie et al 2021)

Deep Cluster [Code]

- - DeepCluster is a method that combines deep learning with clustering to perform unsupervised feature learning. It leverages the idea of clustering learned embeddings to create pseudo-labels and subsequently trains a neural network on these pseudo-labels.
  - Core Steps:
    1. Training a deep neural network.
    2. Applying a clustering algorithm to the embeddings.
    3. Using the cluster assignments as pseudo-labels to train a deep network in a semi-supervised manner.

Multiview Coding

Apply the InfoNCE objective to two or more different views of input data
Became a mainstream contrastive learning method:
- AMDIM (Bachman et al. 2019)
- Contrastive Multiview Coding (CMC; Tian et al. 2019)

Contrastive Learning Between Modalities

“Views” can be from paired inputs from two or more modalities
- CLIP (Radford et al. 2021), ALIGN (Jia et al. 2021): enables zero-shot classification, cross-modal retrieval, guided image generation.
- CodeSearchNet (Husain et al 2019): contrast learning between text and code.

Pretext Tasks

Image Pretext Tasks

Variational Autoencoders: Auto-Encoding Variational Bayes (Kingma et al. 2014)

Generative Adversarial Networks: Jointly train an encoder, additional to the usual GAN (Goodfellow et al. 2014):
1. Bidirectional GAN (BiGAN; Donahue et al. 2016)
2. Adversarially Learned Inference (ALI; Dumoulin et al. 2016)

Bidirectional GAN (BiGAN; Donahue et al. 2016)

- - A BiGAN, or Bidirectional GAN, is a type of generative adversarial network where the generator not only maps latent samples to generated data, but also has an inverse mapping from data to the latent representation. The motivation is to make a type of GAN that can learn rich representations for us in applications like unsupervised learning.
  - In addition to the generator G from the standard GAN framework, BiGAN includes an encoder E which maps data x to latent representations z. The BiGAN discriminator D discriminates not only in data space (x versus G(z)), but jointly in data and latent space (tuples (x, E(x)) versus (G(z),z)), where the latent component is either an encoder output E(x) or a generator input z.

Vision Pretext Tasks

Autoregressive Image Generation:
- Neural autoregressive density estimation (NADE; Larochelle et al. 2011)
- PixelRNN, PixelCNN (Oord et al. 2016)
- Image GPT (Chen et al. 2020)

PixelRNN, PixelCNN (Oord et al. 2016)

- 1. - A PixelCNN is a generative model that uses autoregressive connections to model images pixel by pixel, decomposing the joint image distribution as a product of conditionals. PixelCNNs are much faster to train than PixelRNNs because convolutions are inherently easier to parallelize; given the vast number of pixels present in large image datasets this is an important advantage.

Image Generation:
- Diffusion modeling: Follows a Markov chain of diffusion steps to slowly add random noise to data and then learn to reverse the diffusion process to construct desired data samples from the noise. (Sohl-Dickstein et al 2015; Yang & Ermon 2019; Ho et al. 2020; Dhariwal & Nichol 2021)

Masked Prediction:
1. Denoising Autoencoder (Vincent et al. 2008)
  - Add noise = Randomly mask some pixels
  - Only reconstruction loss
2. Context Autoencoder (Pathak et al. 2016)
  - Mask a random region in the image
  - Reconstruction loss + adversarial loss

- Colorization and More:
  1. Colorization (Zhang et al. 2016)
    - Predict the binned CIE Lab color space given a grayscale image.
  2. Split-brain autoencoder (Zhang et al. 2017)
    - Predict a subset of color channels from the rest of channels.
    - Channels: luminosity, color, depth, etc.

To solve this task, the model has to learn about different objects present in the image and related parts so that it can paint those parts in the same color. Thus, representations learned are useful for downstream tasks.

Innate Relationship Prediction: Learn the relationship among image patches
1. Predict relative positions between patches (Doersch et al 2015)
2. Jigsaw puzzle using patches (Noroozi & Favaro 2016)
3. RotNet: predict which rotation is applied (Gidaris et al. 2018) - Rotation does not alter the semantic content of an image.
4. Representation Learning by Learning to Count (Noroozi et al. 2017) - Counting features across patches without labels, using equivariance of counts

Contrastive Predictive Coding and InfoNCE:
- Contrastive Predictive Coding (CPC) (van den Oord et al. 2018) - classify the “future” representation amongst a set of unrelated “negative” samples

Inter-Sample Classification:
- Exemplar CNN (Dosovitskiy et al. 2014)
- Instance-level discrimination (InstDisc; Wu et al. 2018)

Contrastive Learning: The common approach is to make multiple views (e.g. data augmentation) to one image and consider the image and its distorted version as similar pairs, while different images are treated dissimilar.

Data Augmentation and Multiple Views:
- Augmented Multiscale Deep InfoMax (AMDIM; Bachman et al. 2019) - Views from different augmentations.
- Contrastive Multiview Coding (CMC; Tian et al. 2019) - Multiple views from different channels.
- Pretext-Invariant Representation Learning (PIRL; Misra et al. 2019) - Jigsaw transformation

Inter-Sample Classification: Momentum Contrast (MoCo; He et al. 2019)
- Memory bank is a FIFO queue now.
- The target features are encoded using a momentum encoder.
- MoCo v2 (Chen et al. 2020): MLP projection head & stronger data augmentation.
- MoCo v3 (Chen et al. 2021): Vision Transformer, in-batch negatives.

Inter-Sample Classification: SimCLR (Simple framework for Contrastive Learning of visual Representations)
1. Contrastive learning loss.
2. SimCLR (Chen et al. 2020 Feb)
  - f(.) - base encoder
  - g(.) - projection head layer
  - In-batch negative samples
3. SimCLRv2 (Chen et al. 2020 Jun)
  - Larger ResNet models
  - Deeper g(.)
  - Memory bank

Inter-Sample Classification: Barlow Twins (Zbontar et al. 2021) - Learn to make the cross-correlation matrix between two output features for two distorted version of the same sample close to the identity.

Non-Contrastive Siamese Networks: Learn similarity representations for different augmented views of the same sample, but no contrastive component involving negative samples
- Minimize L2 distance between online and target features
- Bootstrap Your Own Latent (BYOL; Grill et al. 2020) - Momentum-encoded features as the target
- SimSiam (Chen & He 2020) - No momentum encoder and Large batch size unnecessary
- BatchNorm seems to be playing an important role

Feature Clustering with K-Means:
- DeepCluster (Caron et al. 2018): Iteratively clusters features via k-means and uses cluster assignments as pseudo labels to provide supervised signals.
- Online DeepCluster (Zhan et al. 2020): Performs clustering and network update simultaneously rather than alternatingly.
- Prototypical Cluster Learning (PCL; Li et al. 2020): Online EM for clustering, combined with InfoNCE for smoothness

Feature Clustering with Sinkhorn-Knopp:
- SeLa (Self-Labelling; Asano et al. 2020)
- SwAV (Swapping Assignments between multiple Views; Caron et al. 2020) - Implicit clustering via a learned prototype code (“anchor clusters”); and Predict cluster assignment in the other column.

Feature Clustering to Improve SSL:
- InterCLR (Xie et al. 2020): Inter-sample contrastive pairs are constructed according to pseudo labels obtained by clustering.
- Divide and Contrast (Tian et al. 2021): Train expert models on the clustered datasets and then distill the experts into a single model.

Nearest-Neighbor: NNCLR (Dwibedi et al. 2021)
- Contrast with the nearest neighbors in the embedding space
- Allows for lighter data augmentation for views

- Combining with Supervised Loss:

1. Combine supervised loss + self-supervised learning

(1) Self-supervised semi-supervised learning (S4L; Zhai et al 2019)

(2) Unsupervised data augmentation (UDA; Xie et al 2019)

2. Use known labels for contrastive learning

(1) Supervised Contrastive Loss (SupCon; Khosla et al. 2021)

Video Pretext Tasks

Innate Relationship Prediction: Most image pretext tasks can be applied to videos. However, with an additional time dimension, much more information about the video shot configuration or the physical world can be extracted from videos.
1. Predicting object movements.
2. 3D motion of camera.

Optical Flow: Tracking object movement tracking in time.
1. Tracking movement of image patches (Wang & Gupta 2016)
2. Segmenting based on motion (Pathak et al. 2017)

Sequence Ordering:
1. Temporal order verification (Misra et al. 2016, Fernando et al. 2017)
2. Predict the arrow of time, forward or backward (Wei et al. 2018)

Colorization: Tracking emerges by colorizing videos (Vondrick et al. 2018)
1. Copy colors from a reference frame to another target frame in grayscale by leveraging the natural temporal coherence of colors across video frames.
2. Used for video segmentation or human pose estimation without fine-tuning!

Contrastive Multi-View Learning
1. TCN (Sermanet et al. 2017) - Use triplet loss + Different viewpoints at the same time step of the same scene should share the same embedding, while embedding should vary in time, even of the same camera viewpoint.
2. Multi-frame TCN (Dwibedi et al. 2019) - Use n-pairs loss + Multiple frames are aggregated into one embedding.

Autoregressive Generation
1. Predicting videos with VQ-VAE (Walker et al. 2021)
2. VideoGPT: Video generation using VQ-VAE and Transformers (Yan et al. 2021)

Audio Pretext Tasks

Autoregressive Modeling
1. Jukebox (Dhariwal et al. 2020)

Autoregressive Modeling
- CALM (Castellon et al. 2021) - Jukebox representation for MIR tasks
- TagBox (Manilow et al. 2021) - Source separation by steering Jukebox’ latent space

CALM (Castellon et al. 2021)

TagBox (Manilow et al. 2021)

Contrastive Learning
1. COLA (Saeed et al. 2021) assigns high similarity between audio clips extracted from the same recording and low similarity to clips from different recordings.
2. Multi-Format audio contrastive learning (Wang & van den Oord 2021) assigns high similarity between the raw audio format and the corresponding spectral representation

Masked Language Modeling for ASR
1. Wav2Vec 2.0 (Baevski et al. 2020)
2. HuBERT (Hsu et al. 2021)

Multimodal Pretext Tasks

MIL-NCE (Miech et al. 2020)
- Find matching narration with video
CLIP (Radford et al. 2021), ALIGN (Jia et al. 2021)
- Contrast text and image embeddings from paired data

Language Pretext Tasks

Pre-trained language models: They all rely on unsupervised text and try to predict one sentence from the context.

GPT: Autoregressive; predict the next token based on the previous tokens.
BERT: Masked language modeling (MLM) & Next sentence prediction (NSP).
ALBERT: Sentence order prediction (SOP).
ELECTRA: Replaced token detection (RTD).

Sentence Embedding:
1. Skip-thought vectors (Kiros et al. 2015) - Predict sentences based on other sentences around.
2. Quick-thought vectors (Logeswaran & Lee 2018) - Identify the correct context sentence among other contrastive sentences.

Sentence Embedding:
1. IS-BERT (“Info-Sentence BERT”; Zhang et al. 2020) - mutual information maximization
2. SimCSE (“Simple Contrastive learning of Sentence Embeddings”; Gao et al. 2021) - Predict a sentence from itself with only dropout noise; One sentence gets two different versions of dropout augmentations.
3. Most of the models for learning sentence embedding relies on supervised NLI (Natural Language Inference) datasets, such as SBERT (Reimers & Gurevych 2019), BERT-flow (Li et al. 2020), Whitening SBERT (Su et al. 2021).
4. Unsupervised sentence embedding models (e.g. unsupervised SimCSE) still have performance gap with the supervised version (e.g. supervised SimCSE).

Techniques

Data augmentation
In-batch negative samples
Hard negative mining
Memory bank
Large batch size

Data Augmentation

- 1. Data augmentation setup is critical for learning good embedding.
  2. It introduces the non-essential variations into examples without modifying semantic meanings and thus encourages the model to learn the essential part within the representation.

● Image augmentation

● Text augmentation

Image Augmentation

Basic Image Augmentation
- Random crop.
- Color distortion.
- Gaussian blur.
- Color jittering.
- Random flip/rotation.
Augmentation Strategies
- AutoAugment (Cubuk, et al. 2018): inspired by NAS
- RandAugment (Cubuk et al. 2019): reduces NAS search space in AutoAugment.
- PBA (Population based augmentation; Ho et al. 2019): evolutionary algorithm
- UDA (Unsupervised Data Augmentation; Xie et al. 2019): minimize the KL divergence between the predicted distribution over an unlabelled example and its unlabelled augmented version.
Image Mixture
- Mixup (Zhang et al 2018): weighted pixel-wise combination of two images.
- Cutmix (Yun et al 2019): mix in a local region of one image into the other.
- MoCHi (“Mixing of Contrastive Hard Negatives”; Kalantidis et al 2020): mixture of hard negative samples.

Text Augmentation

- 1. - 1. Lexical Edits
        EDA (Easy Data Augmentation; Wei & Zou 2019): synonym replacement, random insertion/swap/deletion.
        Contextual Augmentation (Kobayashi 2018): word substitution by BERT prediction.
      2. Back-translation
        Back-translation augments one sentence by first translating it to another language and then translating it back to the original language.
        CERT (Fang et al. 2020) generates augmented sentences via back-translation.
      3. Dropout and Cutoff:
        SimCSE uses dropout (Gao et al. 2021)
        Cutoff augmentation for text (Shen et al. 2020): tokens, feature columns, spans.

Hard Negative Mining

Hard negative samples are different to learn. They should have different labels from the anchor sample, but the embedding features may be very close. Challenging negative samples encourages the model to learn better representations that can distinguish hard negatives from true positives.

(1) Explicit hard negative mining:

- 1. 1. Extract task-specific hard negative samples from labelled datasets. e.g. “contradiction” sentence pairs from NLI datasets. (Most sentence embedding papers)
    2. Keyword based retrieval. e.g. BM25 search results (Karpukhin et al. 2020)
    3. Upweight the negative sample probability to be proportional to its similarity to the anchor sample (Robinson et al. 2021)
    4. MoCHi (Kalantidis et al. 2020): mine hard negative by sorting them according to similarity to the query in descending order.

(2) Implicit hard negative mining:

In-batch negative samples.
Memory bank (Wu et al. 2018, He et al. 2019) → Increase batch size.
Large batch size via various training parallelism.

Future Directions

Large batch size → improved transfer performance.
High-quality large data corpus → Better performance.
1. Learning from synthetic or Web data.
2. Measuring dataset quality and filtering / active learning
Efficient negative sample selection.
Combine multiple pretext tasks.
1. How to combine
2. Best strategies

Data augmentation tricks have critical impacts but are still quite ad-hoc
1. Modality-dependent
2. Theoretical foundations
Improving training efficiency
1. Self-supervised learning methods are pushing the deep learning arms race
2. Direct impacts on the economical and environmental costs
Social biases in the embedding space.
1. Early work in debiasing word embedding.
2. Biases in Dataset

Challenge in SSL

Accuracy:
- Although the premise of the SSL technique is to not use labelled data, the downside to that approach is you either need huge amounts of data to generate accurate pseudo labels or compromise on accuracy.
- It is important to note that inaccurate labels generated will be counterproductive while training in the initial steps.
Computational Efficiency:
- Due to multiple stages of training (1. Generating pseudo labels 2. Training on pseudo labels) the time taken to train a model is high as compared to supervised learning.
- Current SSL approaches require a huge amount of data to achieve accuracy close to supervised learning counterparts.
Pretext Task:
- It is very important to choose the right pretext task for your use case.
- For instance, if you choose an autoencoder as your pretext task where the image is compressed and then regenerated, it will also try to mimic the noise of the original image and if your task is generating high-quality images, this pretext task will do more harm than good.

References

Page updated

Google Sites

Report abuse