[Paper Review] The Impact of Isolation Kernel on Agglomerative Hierarchical Clustering Algorithms

Xin Han, Ye Zhu+2 more|arXiv (Cornell University)|Oct 12, 2020

Advanced Clustering Algorithms Research4 citations

TL;DR

This paper proposes replacing distance-based linkage in agglomerative hierarchical clustering (AHC) with the Isolation Kernel—a data-dependent kernel that adapts to local density—to address the bias in traditional AHC that merges dense clusters first, leading to poor dendrogram quality on datasets with varied-density clusters. Empirical evaluation shows the Isolation Kernel significantly improves dendrogram purity across multiple AHC algorithms, outperforming distance, Gaussian Kernel, and adaptive Gaussian Kernel.

ABSTRACT

Agglomerative hierarchical clustering (AHC) is one of the popular clustering approaches. Existing AHC methods, which are based on a distance measure, have one key issue: it has difficulty in identifying adjacent clusters with varied densities, regardless of the cluster extraction methods applied on the resultant dendrogram. In this paper, we identify the root cause of this issue and show that the use of a data-dependent kernel (instead of distance or existing kernel) provides an effective means to address it. We analyse the condition under which existing AHC methods fail to extract clusters effectively; and the reason why the data-dependent kernel is an effective remedy. This leads to a new approach to kernerlise existing hierarchical clustering algorithms such as existing traditional AHC algorithms, HDBSCAN, GDL and PHA. In each of these algorithms, our empirical evaluation shows that a recently introduced Isolation Kernel produces a higher quality or purer dendrogram than distance, Gaussian Kernel and adaptive Gaussian Kernel.

Motivation & Objective

• To identify and formally analyze the root cause of poor clustering performance in traditional agglomerative hierarchical clustering (T-AHC) when dealing with clusters of varied densities.
• To demonstrate that the bias in T-AHC—favoring early merging of high-density regions—is due to the use of data-independent distance or kernel measures.
• To propose a generic, plug-in solution using the Isolation Kernel to replace distance in existing AHC algorithms without modifying their core linkage functions.
• To evaluate the effectiveness of the Isolation Kernel in improving dendrogram purity compared to distance, Gaussian Kernel, and adaptive Gaussian Kernel across multiple AHC algorithms.
• To establish formal metrics—entanglement count and average entanglement level—that correlate strongly with dendrogram purity, enabling objective evaluation of clustering quality.

Proposed method

• Introduce the concept of 'entanglement' to quantify the severity of cross-cluster merging in dendrograms, defined as subtrees containing points from different true clusters.
• Define two entanglement indicators: the number of entanglements and the average entanglement level, which are shown to correlate highly with dendrogram purity.
• Propose the Isolation Kernel as a data-dependent kernel that adapts to local density, reducing the bias toward merging dense clusters prematurely.
• Apply the Isolation Kernel as a replacement for distance in four existing AHC algorithms: single-linkage AHC, complete-linkage AHC, HDBSCAN, GDL, and PHA, without modifying their core algorithms.
• Use the distance matrix as input for fair complexity comparison, ensuring consistent computational evaluation across all methods.
• Conduct a Friedman test with post-hoc Nemenyi test to assess statistical significance of performance differences in F1 scores across kernel types.

Experimental results

Research questions

• RQ1 Under what formal conditions does a traditional AHC fail to extract true clusters effectively when clusters have varied densities?
• RQ2 Why does the use of data-independent distance or kernel measures lead to a bias in merging dense clusters before sparse ones in AHC?
• RQ3 How does the Isolation Kernel, as a data-dependent kernel, mitigate the density bias in AHC algorithms?
• RQ4 To what extent does replacing distance with the Isolation Kernel improve dendrogram purity across multiple AHC algorithms?
• RQ5 Are the entanglement indicators (number and average level) statistically correlated with dendrogram purity as an objective measure?

Key findings

• The Isolation Kernel significantly improves dendrogram purity compared to distance, Gaussian Kernel, and adaptive Gaussian Kernel across all tested AHC algorithms.
• The post-hoc Nemenyi test shows that the Isolation Kernel is statistically superior to all other measures at α=0.10, with no significant difference between the other kernels.
• The number of entanglements and average entanglement level are highly correlated with dendrogram purity, validating their use as diagnostic indicators of clustering quality.
• The Isolation Kernel’s runtime is slightly higher than other kernels due to its ensemble nature, but it is amenable to GPU acceleration, mitigating performance concerns.
• The proposed method is generic: replacing distance with the Isolation Kernel improves clustering quality in existing AHC algorithms without modifying their linkage functions or algorithmic structure.
• Empirical results confirm that the Isolation Kernel outperforms existing kernels on four real-world datasets (WDBC, Banknote, Segment, Spam), with consistent F1 score improvements across all algorithms.