[Paper Review] Image Clustering with External Guidance

Yunfan Li, Peng Hu+4 more|arXiv (Cornell University)|Oct 18, 2023

Image Retrieval and Classification Techniques4 citations

TL;DR

This paper proposes Text-Aided Clustering (TAC), a novel externally guided image clustering method that leverages WordNet's textual semantics to enhance feature discriminability and improve clustering performance. By jointly distilling neighborhood information between images and text, TAC achieves state-of-the-art results on five standard and three challenging benchmarks, including full ImageNet-1K, demonstrating the effectiveness of external knowledge as a supervision signal.

ABSTRACT

The core of clustering is incorporating prior knowledge to construct supervision signals. From classic k-means based on data compactness to recent contrastive clustering guided by self-supervision, the evolution of clustering methods intrinsically corresponds to the progression of supervision signals. At present, substantial efforts have been devoted to mining internal supervision signals from data. Nevertheless, the abundant external knowledge such as semantic descriptions, which naturally conduces to clustering, is regrettably overlooked. In this work, we propose leveraging external knowledge as a new supervision signal to guide clustering, even though it seems irrelevant to the given data. To implement and validate our idea, we design an externally guided clustering method (Text-Aided Clustering, TAC), which leverages the textual semantics of WordNet to facilitate image clustering. Specifically, TAC first selects and retrieves WordNet nouns that best distinguish images to enhance the feature discriminability. Then, to improve image clustering performance, TAC collaborates text and image modalities by mutually distilling cross-modal neighborhood information. Experiments demonstrate that TAC achieves state-of-the-art performance on five widely used and three more challenging image clustering benchmarks, including the full ImageNet-1K dataset.

Motivation & Objective

• To address the limitation of internal supervision signals in clustering, which are bounded by data complexity and distribution.
• To explore and validate external knowledge—specifically textual semantics from WordNet—as a novel, underutilized supervision signal for image clustering.
• To design a method that effectively integrates external text semantics with image features to improve clustering discriminability and performance.
• To demonstrate that external knowledge can yield significant gains even when seemingly unrelated to the input data.

Proposed method

• TAC constructs a text space by selecting and retrieving WordNet nouns that best distinguish image semantic centers, enhancing feature discriminability.
• It formulates a retrieval process using a softness temperature to generate discriminative text counterparts for image clusters.
• The method performs mutual distillation of neighborhood information between image and text modalities to align and refine clustering representations.
• It employs a balance loss with a learnable weight to prevent cluster collapse and ensure even cluster distribution.
• A cluster number estimation criterion is introduced to automatically determine the optimal number of clusters based on semantic center distribution.
• The model jointly optimizes clustering objectives using contrastive learning and mutual distillation to improve representation quality.

Figure 1: The evolution of clustering methods could be roughly divided into three eras, including i) classic clustering , which designs clustering strategies based on data distribution assumptions; ii) deep clustering , which extracts clustering-favorable features with deep neural networks by formul

Experimental results

Research questions

• RQ1 Can external textual knowledge from pre-existing knowledge bases like WordNet serve as an effective supervision signal for image clustering?
• RQ2 How can textual semantics be effectively aligned with image features to improve clustering performance?
• RQ3 What hyperparameters govern the performance of the text-augmented clustering framework, and how should they be tuned?
• RQ4 Does leveraging external knowledge outperform traditional self-supervised or internal supervision-based clustering methods?

Key findings

• TAC achieves state-of-the-art performance on five widely used image clustering benchmarks, including full ImageNet-1K, demonstrating significant improvements over prior methods.
• The method shows robustness across diverse datasets, with performance stable across different values of key hyperparameters like the number of nearest neighbors and mutual distillation temperature.
• The optimal number of discriminative nouns per cluster lies in the range of 3 to 10, with performance degrading for both smaller and larger values.
• The softness temperature in noun retrieval is critical: too low causes collapse into identical nouns, while too high mixes semantics across different image classes.
• The cluster number estimation criterion in Eq. 2 effectively identifies the optimal number of clusters, improving performance on both ImageNet-Dogs and UCF-101.
• The balance loss weight α = 5 was found to be effective across all eight datasets tested, preventing model collapse and ensuring balanced cluster assignments.

Figure 2: The observations of this paper. As a showcase, two image examples from the ImageNet-Dogs dataset are illustrated. For each example, we show the manually annotated class names and the nouns obtained by the proposed TAC, as well as the zero-shot classification probabilities. From the example