[Paper Review] A New Index for Clustering Evaluation Based on Density Estimation

Gangli Liu|arXiv (Cornell University)|Jul 4, 2022

Advanced Clustering Algorithms Research4 citations

TL;DR

This paper proposes a novel internal clustering evaluation index based on kernel density estimation, combining an Ambiguous Index and a Similarity Index to assess cluster quality. Evaluated on 145 datasets, the new index significantly outperforms six established indices, including Calinski-Harabasz, Silhouette, and Davies-Bouldin, demonstrating superior accuracy in identifying optimal clusterings.

ABSTRACT

A new index for internal evaluation of clustering is introduced. The index is defined as a mixture of two sub-indices. The first sub-index $ I_a $ is called the Ambiguous Index; the second sub-index $ I_s $ is called the Similarity Index. Calculation of the two sub-indices is based on density estimation to each cluster of a partition of the data. An experiment is conducted to test the performance of the new index, and compared with six other internal clustering evaluation indices -- Calinski-Harabasz index, Silhouette coefficient, Davies-Bouldin index, CDbw, DBCV, and VIASCKDE, on a set of 145 datasets. The result shows the new index significantly improves other internal clustering evaluation indices.

Motivation & Objective

• To address the challenge of internal clustering evaluation, where ground truth is unavailable and existing indices often fail to capture complex cluster structures.
• To improve the reliability and accuracy of internal evaluation by incorporating density estimation to model cluster distribution and separation.
• To develop a new index that combines ambiguity and similarity metrics derived from density estimates for more robust cluster quality assessment.
• To empirically validate the new index against six prominent internal evaluation indices across diverse real-world and synthetic datasets.
• To explore variants of the index and assess the impact of boundary detection and density estimation techniques on performance.

Proposed method

• The proposed index is a weighted mixture of two sub-indices: the Ambiguous Index ($I_a$), which quantifies overlap between cluster density regions, and the Similarity Index ($I_s$), which measures intra-cluster homogeneity.
• Density estimation is performed per cluster using kernel density estimation (KDE), enabling probabilistic modeling of cluster structure and boundary detection.
• The Ambiguous Index is computed by evaluating pairwise density overlap between clusters, with variants using proportion of positive overlaps or mean of positive overlaps.
• The Similarity Index is derived from log-likelihood values of data points within each cluster, normalized and aggregated to reflect cluster compactness.
• A Boundary Index ($I_b$) is introduced to detect points near cluster edges using a thresholded density range based on cluster-specific standard deviation.
• The final index is a combination of $I_a$ and $I_s$, with preliminary tests showing that including $I_b$ offers marginal improvement without significant complexity gain.

Experimental results

Research questions

• RQ1 Can a density-based internal evaluation index outperform traditional indices like Calinski-Harabasz and Silhouette in identifying optimal clusterings?
• RQ2 How does incorporating both ambiguity (inter-cluster overlap) and similarity (intra-cluster compactness) improve clustering evaluation?
• RQ3 What is the impact of different variants of the Ambiguous and Similarity sub-indices on overall performance?
• RQ4 Does the inclusion of a boundary detection mechanism enhance the index’s ability to identify poor clusterings?
• RQ5 How does the new index perform across diverse datasets with varying cluster shapes, densities, and noise levels?

Key findings

• The proposed index significantly outperforms six established internal clustering evaluation indices—Calinski-Harabasz, Silhouette, Davies-Bouldin, CDbw, DBCV, and VIASCKDE—on a benchmark of 145 datasets.
• The new index achieves higher performance in identifying optimal clusterings, particularly in cases with overlapping or non-spherical clusters where traditional indices fail.
• Variants of the Ambiguous and Similarity indices, such as $I_{a\_v1}$, $I_{a\_v2}$, and $I_{s\_v1}$, yield comparable performance to the main index, indicating robustness to parameterization.
• The Boundary Index ($I_b$) shows potential for improving detection of weakly defined clusters, though its inclusion only marginally improves overall performance.
• The index is computationally more intensive than non-density-based indices due to per-cluster KDE fitting, but performance can be accelerated via parallelization.
• Preliminary results suggest the index maintains strong performance across diverse data distributions, though its behavior in high-dimensional spaces and under noise requires further investigation.