[Paper Review] Exploring Non-Contrastive Representation Learning for Deep Clustering

Zhizhong Huang, Jie Chen+2 more|arXiv (Cornell University)|Nov 23, 2021

Domain Adaptation and Few-Shot Learning53 references4 citations

TL;DR

This paper proposes NCC, a non-contrastive deep clustering method based on BYOL that eliminates negative examples to avoid class collision. By introducing a positive sampling strategy and prototypical contrastive loss within an EM framework, NCC enhances within-cluster compactness and inter-cluster separation, achieving state-of-the-art performance on ImageNet-1K and other benchmarks.

ABSTRACT

Existing deep clustering methods rely on contrastive learning for representation learning, which requires negative examples to form an embedding space where all instances are well-separated. However, the negative examples inevitably give rise to the class collision issue, compromising the representation learning for clustering. In this paper, we explore non-contrastive representation learning for deep clustering, termed NCC, which is based on BYOL, a representative method without negative examples. First, we propose to align one augmented view of instance with the neighbors of another view in the embedding space, called positive sampling strategy, which avoids the class collision issue caused by the negative examples and hence improves the within-cluster compactness. Second, we propose to encourage alignment between two augmented views of one prototype and uniformity among all prototypes, named prototypical contrastive loss or ProtoCL, which can maximize the inter-cluster distance. Moreover, we formulate NCC in an Expectation-Maximization (EM) framework, in which E-step utilizes spherical k-means to estimate the pseudo-labels of instances and distribution of prototypes from a target network and M-step leverages the proposed losses to optimize an online network. As a result, NCC forms an embedding space where all clusters are well-separated and within-cluster examples are compact. Experimental results on several clustering benchmark datasets including ImageNet-1K demonstrate that NCC outperforms the state-of-the-art methods by a significant margin.

Motivation & Objective

• To address the class collision issue inherent in contrastive learning-based deep clustering, which arises due to the use of negative examples.
• To explore non-contrastive representation learning as a viable alternative to contrastive methods for deep clustering.
• To improve within-cluster compactness and inter-cluster separation without relying on negative mining.
• To develop a novel training framework that integrates pseudo-labeling and prototype learning within an EM optimization scheme.

Proposed method

• Proposes a positive sampling strategy that aligns one augmented view of an instance with neighbors of another augmented view in the embedding space, avoiding negative examples.
• Introduces a prototypical contrastive loss (ProtoCL) that encourages alignment between two augmented views of the same prototype and uniformity across all prototypes.
• Employs an Expectation-Maximization (EM) framework: E-step uses spherical k-means on the target network's output to estimate pseudo-labels and prototype distributions.
• M-step optimizes the online network using the proposed positive sampling and ProtoCL losses to refine representations.
• Leverages a target network to stabilize training and improve pseudo-label quality during the E-step.
• Formulates the entire framework to maximize both within-cluster compactness and inter-cluster separation in the embedding space.

Experimental results

Research questions

• RQ1 Can non-contrastive representation learning eliminate the class collision problem present in contrastive learning for deep clustering?
• RQ2 How can within-cluster compactness be improved without relying on negative examples?
• RQ3 Can a prototype-level contrastive loss enhance inter-cluster separation in deep clustering?
• RQ4 Does integrating non-contrastive learning into an EM framework lead to better clustering performance?

Key findings

• NCC outperforms state-of-the-art contrastive and non-contrastive deep clustering methods on multiple benchmark datasets.
• The proposed positive sampling strategy significantly reduces the class collision issue by avoiding negative examples.
• The prototypical contrastive loss (ProtoCL) effectively increases inter-cluster distances, improving cluster separability.
• The EM-based framework with spherical k-means in the E-step leads to more accurate pseudo-labels and stable training.
• On ImageNet-1K, NCC achieves a significant performance margin over existing methods, demonstrating strong generalization.
• The ablation study confirms the effectiveness of both the positive sampling and ProtoCL components in boosting clustering accuracy.