[Paper Review] Deep Robust Clustering by Contrastive Learning

Huasong Zhong, Chong Chen+2 more|arXiv (Cornell University)|Aug 7, 2020

Domain Adaptation and Few-Shot Learning39 references43 citations

TL;DR

DRC introduces a one-stage deep clustering method that jointly optimizes assignment probability and assignment feature through contrastive learning, improving clustering stability and accuracy over state-of-the-art methods. It frames mutual information via two contrastive losses and a cluster-regularization term.

ABSTRACT

Recently, many unsupervised deep learning methods have been proposed to learn clustering with unlabelled data. By introducing data augmentation, most of the latest methods look into deep clustering from the perspective that the original image and its transformation should share similar semantic clustering assignment. However, the representation features could be quite different even they are assigned to the same cluster since softmax function is only sensitive to the maximum value. This may result in high intra-class diversities in the representation feature space, which will lead to unstable local optimal and thus harm the clustering performance. To address this drawback, we proposed Deep Robust Clustering (DRC). Different from existing methods, DRC looks into deep clustering from two perspectives of both semantic clustering assignment and representation feature, which can increase inter-class diversities and decrease intra-class diversities simultaneously. Furthermore, we summarized a general framework that can turn any maximizing mutual information into minimizing contrastive loss by investigating the internal relationship between mutual information and contrastive learning. And we successfully applied it in DRC to learn invariant features and robust clusters. Extensive experiments on six widely-adopted deep clustering benchmarks demonstrate the superiority of DRC in both stability and accuracy. e.g., attaining 71.6% mean accuracy on CIFAR-10, which is 7.1% higher than state-of-the-art results.

Motivation & Objective

• Motivate and address instability in deep clustering due to intra-class diversity when using assignment probabilities alone.
• Propose a framework that learns both invariant cluster assignments and invariant features via contrastive learning.
• Show that maximizing mutual information can be transformed into minimizing tractable contrastive losses.
• Demonstrate robustness and accuracy gains across multiple standard clustering benchmarks.

Proposed method

• Define assignment feature z and assignment probability p via a CNN and softmax.
• Maximize mutual information between original and augmented samples in two views: assignment probability (global view) and assignment feature (local view).
• Convert mutual information objectives into two contrastive losses: L_AF (assignment feature) and L_AP (assignment probability).
• Add a cluster regularization loss L_CR to prevent trivial solutions and stabilize training.
• Train end-to-end with Adam, using a ResNet-based backbone, batch size 256, and augmentation pairs; hyperparameters include T=0.5 for AF and T=0.95 for AP, λ=0.005.

Experimental results

Research questions

• RQ1 Can a deep clustering model improve both inter-class separation and intra-class compactness by considering both assignment probability and assignment feature?
• RQ2 Is it possible to reinterpret mutual information maximization as a set of practical contrastive losses for stable training?
• RQ3 How does DRC perform on standard unsupervised clustering benchmarks relative to prior methods?
• RQ4 What is the impact of ablations on the two contrastive losses and the cluster regularization term?

Key findings

• DRC achieves state-of-the-art or near state-of-the-art clustering performance on six benchmarks, with substantial gains in mean accuracy over baselines and prior methods.
• DRC demonstrates reduced variance in clustering results across initializations, indicating improved stability.
• Both contrastive losses (AP and AF) contribute to performance, with AP generally having a larger direct effect on clustering accuracy.
• Cluster regularization (L_CR) improves results and mitigates trivial solutions, especially on harder datasets.
• Larger batch sizes enhance contrastive learning performance and final clustering metrics.
• Intra-class variance is reduced and inter-class variance increased under DRC compared to prior methods.