[论文解读] Clustering-friendly Representation Learning via Instance Discrimination and Feature Decorrelation

Yaling Tao, Kentaro Takagi+1 more|arXiv (Cornell University)|May 31, 2021

Domain Adaptation and Few-Shot Learning参考文献 35被引用 36

一句话总结

该论文将实例辨别与基于softmax的特征去相关结合，学习便于聚类的表示，将该方法与谱聚类连接起来，并使用简单的k-means在图像聚类上取得强结果。

ABSTRACT

Clustering is one of the most fundamental tasks in machine learning. Recently, deep clustering has become a major trend in clustering techniques. Representation learning often plays an important role in the effectiveness of deep clustering, and thus can be a principal cause of performance degradation. In this paper, we propose a clustering-friendly representation learning method using instance discrimination and feature decorrelation. Our deep-learning-based representation learning method is motivated by the properties of classical spectral clustering. Instance discrimination learns similarities among data and feature decorrelation removes redundant correlation among features. We utilize an instance discrimination method in which learning individual instance classes leads to learning similarity among instances. Through detailed experiments and examination, we show that the approach can be adapted to learning a latent space for clustering. We design novel softmax-formulated decorrelation constraints for learning. In evaluations of image clustering using CIFAR-10 and ImageNet-10, our method achieves accuracy of 81.5% and 95.4%, respectively. We also show that the softmax-formulated constraints are compatible with various neural networks.

研究动机与目标

• Motivate and develop a clustering-friendly representation learning method for unsupervised image clustering.
• Bridge instance discrimination with spectral clustering by enforcing both similarity among instances and decorrelation of features.
• Propose softmax-formulated decorrelation constraints that are stable and compatible with common neural networks.
• Demonstrate state-of-the-art clustering performance on CIFAR-10 and ImageNet-10 using simple k-means on learned embeddings.

提出的方法

• Adopt instance discrimination where each unlabeled instance is treated as its own class, with a softmax-based objective to learn instance-wise similarities.
• Introduce a feature decorrelation term, including a softmax-formulated constraint, to encourage latent feature orthogonality while allowing flexibility (soft decorrelation).
• Combine the instance discrimination loss with the decorrelation loss into a joint objective L_IDFD = L_I + alpha L_F (alpha tuned per setting).
• Provide a variant with a simple orthogonality constraint L_FO and compare with the softmax decorrelation L_F, highlighting stability and practical benefits.
• Connect the approach to spectral clustering by showing how the learned embeddings approximate the Laplacian eigenspace whenever tau is appropriately set (tau ≥ ~2) and L_F enforces near-orthogonality.
• Evaluate with k-means clustering on learned embeddings across multiple datasets (CIFAR-10, CIFAR-100, STL-10, ImageNet-10, ImageNet-Dog) and networks.

实验结果

研究问题

• RQ1 Can instance-discrimination-based representation learning be tailored to produce clustering-friendly embeddings for unsupervised image clustering?
• RQ2 Does incorporating a softmax-formulated feature decorrelation constraint improve clustering performance and stability compared to strict orthogonality?
• RQ3 How does the temperature parameter tau affect the clustering Quality and the relation to spectral clustering?
• RQ4 Are the proposed IDFD gains robust across different network architectures and datasets?

主要发现

• IDFD (and its variants IDFO, ID(tuned)) consistently outperform several baselines (AE, DEC, DAC, DCCM, IIC, SCAN) on multiple datasets according to ACC, NMI, and ARI.
• On CIFAR-10, IDFD achieves 81.5% ACC, 71.1 NMI, 66.3 ARI; on CIFAR-100 42.5 ACC, 42.6 NMI, 26.4 ARI; STL-10 75.6 ACC, 64.3 NMI, 57.5 ARI; ImageNet-10 95.4 ACC, 89.8 NMI, 90.1 ARI; ImageNet-Dog 59.1 ACC, 54.6 NMI, 41.3 ARI.
• IDFO and IDFD consistently improve over ID(original) across datasets, demonstrating the benefit of feature decorrelation and the larger-tau instance discrimination setting (ID tuned).
• The softmax decorrelation constraint L_F provides stability and flexibility over strict orthogonality (L_FO), with derivatives guiding smoother optimization.
• IDFD works across multiple architectures (ConvNet, VGG16, ResNet18/34) and remains effective when combined with data augmentation, indicating practical applicability.