[Paper Review] Clustering-friendly Representation Learning via Instance Discrimination and Feature Decorrelation

Yaling Tao, Kentaro Takagi+1 more|arXiv (Cornell University)|May 31, 2021

Domain Adaptation and Few-Shot Learning35 references36 citations

TL;DR

The paper blends instance discrimination with softmax-based feature decorrelation to learn clustering-friendly representations, connecting the approach to spectral clustering and achieving strong image clustering results using simple k-means.

ABSTRACT

Clustering is one of the most fundamental tasks in machine learning. Recently, deep clustering has become a major trend in clustering techniques. Representation learning often plays an important role in the effectiveness of deep clustering, and thus can be a principal cause of performance degradation. In this paper, we propose a clustering-friendly representation learning method using instance discrimination and feature decorrelation. Our deep-learning-based representation learning method is motivated by the properties of classical spectral clustering. Instance discrimination learns similarities among data and feature decorrelation removes redundant correlation among features. We utilize an instance discrimination method in which learning individual instance classes leads to learning similarity among instances. Through detailed experiments and examination, we show that the approach can be adapted to learning a latent space for clustering. We design novel softmax-formulated decorrelation constraints for learning. In evaluations of image clustering using CIFAR-10 and ImageNet-10, our method achieves accuracy of 81.5% and 95.4%, respectively. We also show that the softmax-formulated constraints are compatible with various neural networks.

Motivation & Objective

• Motivate and develop a clustering-friendly representation learning method for unsupervised image clustering.
• Bridge instance discrimination with spectral clustering by enforcing both similarity among instances and decorrelation of features.
• Propose softmax-formulated decorrelation constraints that are stable and compatible with common neural networks.
• Demonstrate state-of-the-art clustering performance on CIFAR-10 and ImageNet-10 using simple k-means on learned embeddings.

Proposed method

• Adopt instance discrimination where each unlabeled instance is treated as its own class, with a softmax-based objective to learn instance-wise similarities.
• Introduce a feature decorrelation term, including a softmax-formulated constraint, to encourage latent feature orthogonality while allowing flexibility (soft decorrelation).
• Combine the instance discrimination loss with the decorrelation loss into a joint objective L_IDFD = L_I + alpha L_F (alpha tuned per setting).
• Provide a variant with a simple orthogonality constraint L_FO and compare with the softmax decorrelation L_F, highlighting stability and practical benefits.
• Connect the approach to spectral clustering by showing how the learned embeddings approximate the Laplacian eigenspace whenever tau is appropriately set (tau ≥ ~2) and L_F enforces near-orthogonality.
• Evaluate with k-means clustering on learned embeddings across multiple datasets (CIFAR-10, CIFAR-100, STL-10, ImageNet-10, ImageNet-Dog) and networks.

Experimental results

Research questions

• RQ1 Can instance-discrimination-based representation learning be tailored to produce clustering-friendly embeddings for unsupervised image clustering?
• RQ2 Does incorporating a softmax-formulated feature decorrelation constraint improve clustering performance and stability compared to strict orthogonality?
• RQ3 How does the temperature parameter tau affect the clustering Quality and the relation to spectral clustering?
• RQ4 Are the proposed IDFD gains robust across different network architectures and datasets?

Key findings

• IDFD (and its variants IDFO, ID(tuned)) consistently outperform several baselines (AE, DEC, DAC, DCCM, IIC, SCAN) on multiple datasets according to ACC, NMI, and ARI.
• On CIFAR-10, IDFD achieves 81.5% ACC, 71.1 NMI, 66.3 ARI; on CIFAR-100 42.5 ACC, 42.6 NMI, 26.4 ARI; STL-10 75.6 ACC, 64.3 NMI, 57.5 ARI; ImageNet-10 95.4 ACC, 89.8 NMI, 90.1 ARI; ImageNet-Dog 59.1 ACC, 54.6 NMI, 41.3 ARI.
• IDFO and IDFD consistently improve over ID(original) across datasets, demonstrating the benefit of feature decorrelation and the larger-tau instance discrimination setting (ID tuned).
• The softmax decorrelation constraint L_F provides stability and flexibility over strict orthogonality (L_FO), with derivatives guiding smoother optimization.
• IDFD works across multiple architectures (ConvNet, VGG16, ResNet18/34) and remains effective when combined with data augmentation, indicating practical applicability.