[Paper Review] Classes are not Clusters: Improving Label-based Evaluation of Dimensionality Reduction

Hyeon Jeon, Yun-Hsin Kuo+3 more|arXiv (Cornell University)|Aug 1, 2023

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper introduces Label-Trustworthiness and Label-Continuity (Label-T&C), novel distortion-based measures that evaluate dimensionality reduction (DR) by quantifying changes in cluster-label matching (CLM) between original and embedded spaces, avoiding reliance on the flawed assumption that classes are inherently well-clustered. Label-T&C outperform existing methods in detecting CLM distortions and reveal intrinsic differences in DR techniques across clustering granularities.

ABSTRACT

A common way to evaluate the reliability of dimensionality reduction (DR) embeddings is to quantify how well labeled classes form compact, mutually separated clusters in the embeddings. This approach is based on the assumption that the classes stay as clear clusters in the original high-dimensional space. However, in reality, this assumption can be violated; a single class can be fragmented into multiple separated clusters, and multiple classes can be merged into a single cluster. We thus cannot always assure the credibility of the evaluation using class labels. In this paper, we introduce two novel quality measures -- Label-Trustworthiness and Label-Continuity (Label-T&C) -- advancing the process of DR evaluation based on class labels. Instead of assuming that classes are well-clustered in the original space, Label-T&C work by (1) estimating the extent to which classes form clusters in the original and embedded spaces and (2) evaluating the difference between the two. A quantitative evaluation showed that Label-T&C outperform widely used DR evaluation measures (e.g., Trustworthiness and Continuity, Kullback-Leibler divergence) in terms of the accuracy in assessing how well DR embeddings preserve the cluster structure, and are also scalable. Moreover, we present case studies demonstrating that Label-T&C can be successfully used for revealing the intrinsic characteristics of DR techniques and their hyperparameters.

Motivation & Objective

• Address the fundamental flaw in label-based DR evaluation: the assumption that class labels inherently reflect true cluster structures in high-dimensional data.
• Overcome the risk of misleading conclusions when class labels are fragmented or merged in the original space, which invalidates standard cluster validation measures.
• Develop a robust evaluation framework that quantifies distortions in cluster structure preservation during dimensionality reduction without assuming label validity.
• Enable more accurate and reliable assessment of DR techniques and hyperparameters by measuring how class labels span original and embedded cluster structures.
• Provide a scalable, interpretable method to diagnose the credibility of visual cluster analysis based on DR embeddings.

Proposed method

• Propose Label-Trustworthiness (Label-T) as a measure of CLM degradation: lower scores indicate that points from different classes have moved closer in the embedding than in the original space.
• Define Label-Continuity (Label-C) as a measure of CLM exaggeration: lower scores indicate that points from different classes have moved farther apart in the embedding than in the original space.
• Use clustering validation measures (CVMs) such as DSC and CH_btwn to estimate CLM in both original and embedded spaces, then compute the difference as distortion.
• Frame the evaluation as a distortion-aware process: instead of assuming good CLM in the original space, measure how much CLM changes due to DR.
• Apply CVMs (e.g., DSC, CH_btwn) to compute CLM in both spaces, then compute Label-T and Label-C as the difference in CLM scores.
• Ensure scalability by using efficient CVMs and avoiding expensive pairwise distance computations, enabling use on large datasets.

Experimental results

Research questions

• RQ1 To what extent does the assumption that class labels form coherent clusters in the original space hold in practice?
• RQ2 How can we evaluate the reliability of DR embeddings when class labels may not reflect the true underlying cluster structure?
• RQ3 Can we design a DR evaluation method that detects distortions in cluster structure preservation without relying on the validity of class labels?
• RQ4 How do different DR techniques and hyperparameters affect the preservation of cluster structure across varying levels of clustering granularity?
• RQ5 Can Label-T&C reveal intrinsic characteristics of DR methods that are not captured by standard evaluation metrics like Trustworthiness or KL divergence?

Key findings

• Label-T&C outperform traditional measures like Trustworthiness, Continuity, and KL divergence in detecting both Missing Groups and False Groups distortions in DR embeddings.
• LLE shows high Label-Continuity (low Missing Groups) but low Label-Trustworthiness (high False Groups) at fine granularities, consistent with its local reconstruction focus and poor KL divergence scores.
• t-SNE and UMAP exhibit the worst Label-Continuity scores, indicating they exaggerate cluster separation at global levels, undermining their reliability for global cluster analysis.
• PCA achieves the best Label-T&C scores at coarse granularities, confirming its strength in preserving global cluster structures and aligning with its high KL divergence performance.
• Densmap, a density-preserving variant of UMAP, shows better Label-Continuity than UMAP but worse Label-Trustworthiness, indicating it reduces Missing Groups but increases False Groups, likely due to larger cluster sizes.
• Label-T&C reveal that Densmap better preserves global cluster structure than UMAP, as confirmed by KL divergence, and can detect subtle differences in cluster density and overlap not visible with standard metrics.