Visual Deep Clustering

Clustering is to divide the data set into multiple categories according to the inherent similarity of the data for a large number of unknown labeled datasets, so that the data similarity within the category is larger and the data similarity between the categories is smaller. Belongs to an unsupervised algorithm.

In this work, we focus on reading papers related to the Deep Clustering method and its implementation on a practical project collaborated with a Taiwan company.

Currently, Deep Clustering has some benefits: (1) High Dimensionality, (2) End to End Framework, and (3) Scalability. However, there are still some drawbacks: (1) Hyper-parameters, (2) Lack of interpretability, and (3) Lack of theoritical framework.

Keyword: Unsupervised Representation Learning;

1) Dataset:

2) Papers:

Paper List:

DeepDPM: Deep Clustering With an Unknown Number of Clusters_arXiv_2022 [Paper]
https://deepnotes.io/deep-clustering#direct-cluster-optimization
https://github.com/zhoushengisnoob/DeepClustering
https://academic.oup.com/bib/article/22/1/393/5721075
https://towardsdatascience.com/a-framework-for-contrastive-self-supervised-learning-and-designing-a-new-approach-3caab5d29619

Reading List:

AutoEncoders-based: (Fine-tune the encoder latent space to make representations more clustering friendly)
- Learning Embedding Space for Clustering From Deep Representations_ICBD_2018 [Paper] [Code] [Colab]
- (DEC) Unsupervised Deep Embedding for Clustering Analysis_PMLR_2016 [Paper] [Code]
- (DBC) Discriminatively Boosted Image Clustering_PR_2018 [Paper]
- (DCN) Towards K-means-friendly Spaces - Simultaneous Deep Learning and Clustering_PMRL_2017 [Paper] [Code]
- (DEPICT) Deep Clustering via Joint Convolutional Autoencoder Embedding and Relative Entropy Minimization_ICCV_2017 [Paper] [Code]
Generative Model-based: (Extend the clustering task to generate samples from clusters)
- (VaDE) Variational Deep Embedding - An Unsupervised and Generative Approach to Clustering_IJCAI_2017 [Paper] [Code]
- InfoGAN - Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets_arXiv_2016 [Paper] [Code]
Direct Cluster Optimization: (Only use clustering loss to optimize the DNN)
- (JULE) Joint Unsupervised Learning of Deep Representations and Image Clusters_CVPR_2016 [Paper] [Code]
- (DAC) Deep Adaptive Image Clustering_ICCV_2017 [Paper] [Code]
- (IMSAT) Learning Discrete Representations via Information Maximizing Self-Augmented Training_PMLR_2017 [Paper] [Code]
Survey:
- A Survey of Clustering With Deep Learning From the Perspective of Network Architecture [Paper]
- The challenges of clustering high dimensional data [Paper]
VinAI:

Improving Relational Regularized Autoencoders with Spherical Sliced Fused Gromov Wasserstein_ICLR_2021
Clusformer: A Transformer based Clustering Approach to Unsupervised Large-scale Face and Visual Landmark Recognition_CVPR_2021

To-do List:

Unfilter:

Improving Relational Regularized Autoencoders with Spherical Sliced Fused Gromov Wasserstein_ICLR_2021
C159_Clustering with Deep Learning: Taxonomy and New Methods_ICLR_2018 [Link]
AAVAE: Augmentation-Augmented Variational Autoencoders_2021
C282_Unsupervised Learning of Visual Features by Contrasting Cluster Assignments_arXiv_2020
C161_Clustering with Deep Learning: Taxonomy and New Methods_arXiv_2018
Discriminatively Boosted Image Clustering with Fully Convolutional Auto-Encoders

Application:

C5_Deep soft K-means clustering with self-training for single-cell RNA sequence data_NAR_202

Anamoly Detection

A Unifying Review of Deep and Shallow Anomaly Detection_IEEE_2020 [Paper]

3) Notes:

Overview:

Clustering is one of the fundamental unsupervised method of knowledge discovery. It’s goal is to group similar data points together without supervision or prior knowledge of nature of the clusters. Various aspects of clustering such as distance metrics, feature selection, grouping methods etc. have been extensively studied since the origin of cluster analysis in the 1930s. Because of it’s importance in exploratory understanding of data, clustering has always been an active field of research.

Framework:

Deep clustering algorithms can be broken down into three essential components: deep neural network, network loss, and clustering loss.

Deep Neural Network Architecture:

The deep neural network is the representation learning component of deep clustering algorithms. They are employed to learn low dimensional non-linear data representations from the dataset. Most widely used architectures are autoencoder based, however generative models like Variational Autoencoders[2] and Generative Adversarial Networks[3] have also been used in different algorithms. Variations in network architecture like convolutional neural networks are also widely used.

Loss Functions:

The objective function of deep clustering algorithms are generally a linear combination of unsupervised representation learning loss, here referred to as network loss LR and a clustering oriented loss LC.
They are formulated as: L = λLR + (1−λ)LC.
where λ is a hyperparameter between 0 and 1 that balances the impact of two loss functions.

Network Loss:

The neural network loss refer to the reconstruction loss of autoencoder, variational loss of VAEs, or the adversarial loss of GANs. The network loss is essential for the initialization of the deep neural networks. Usually after a few epochs, the clustering loss is introduced by changing the λ hyperparameter. Some models like JULE[4], DAC[5] and IMSAT[6] discard the network loss altogether in favour of just using clustering loss to guide both representation learning and clustering.

Clustering Loss:

Several different clustering loss has been proposed and used in different algorithms. They can be generally categorized into:
1. Cluster Assignment
  - Cluster assignment losses provides cluster assignments to the data points directly, and no further clustering algorithm is required to be run on top the learnt data representations. Some examples are: k-means loss [7], cluster assignment hardening loss [8] and agglomerative clustering loss[4].
2. Cluster Regularization
  - Cluster regularization loss, on the other hand, only enforces the network to preserve suitable discriminant information from the data in the representations. Further clustering on the representation space is necessary to obtain the clustering result. Some examples are: locality preserving loss, group sparsity loss[9] etc.

References:

Metrics:

Methods:

Dimensionality Reduction:
- https://www.kaggle.com/arthurtok/interactive-intro-to-dimensionality-reduction
- https://www.kaggle.com/tilii7/dimensionality-reduction-pca-tsne
Turn data into DAEta: https://www.kaggle.com/springmanndaniel/1st-place-turn-your-data-into-daeta
Dataset Decomposition: https://www.kaggle.com/shivamb/dataset-decomposition-techniques
Keras Unsupervised Clustering: https://www.dlology.com/blog/how-to-do-unsupervised-clustering-with-keras/
Improving Autoencoder Performance with Pretrained RBMs: https://towardsdatascience.com/improving-autoencoder-performance-with-pretrained-rbms-e2e13113c782
Visualize: https://towardsdatascience.com/visualising-high-dimensional-datasets-using-pca-and-t-sne-in-python-8ef87e7915b
Learning Embedding Space for Clustering From Deep Representations: https://www.youtube.com/watch?v=d8clWY7Y1-o
DeepClustering: https://github.com/Deepayan137/DeepClustering
Pt-DEC: https://github.com/vlukiyanov/pt-dec

Page updated

Google Sites

Report abuse

Visual Deep Clustering

About Me: