[Paper Review] Estimating the Number of Clusters via Normalized Cluster Instability

Jonas M B Haslbeck, Dirk U. Wulff|arXiv (Cornell University)|Aug 26, 2016

Data Analysis with R14 references3 citations

TL;DR

This paper proposes a normalized cluster instability measure that corrects for the influence of cluster size distribution, significantly improving the accuracy of k-means cluster number estimation—especially for large k. The method outperforms unnormalized instability measures and rivals or exceeds distance-based methods like the Gaussian Mixture Model, while model-based and model-free instability approaches show comparable performance.

ABSTRACT

We improve current instability-based methods for the selection of the number of clusters $k$ in cluster analysis by developing a normalized cluster instability measure that corrects for the distribution of cluster sizes, a previously unaccounted driver of cluster instability. We show that our normalized instability measure outperforms current instability-based measures across the whole sequence of possible $k$ and especially overcomes limitations in the context of large $k$. We also compare, for the first time, model-based and model-free approaches to determine cluster-instability and find their performance to be comparable. We make our method available in the R-package \verb+cstab+.

Motivation & Objective

• To address the limitation of existing instability-based methods in selecting the number of clusters k, particularly when k is large or cluster sizes are uneven.
• To correct for the confounding effect of cluster size distribution on instability, which biases existing methods.
• To compare model-based and model-free approaches to cluster instability for the first time in a systematic way.
• To develop and validate a new, normalized instability measure that enhances stability-based k-selection across diverse clustering scenarios.

Proposed method

• Proposes a normalized instability measure, d^n(ψ_a, ψ_b), that adjusts raw clustering distance d^r(ψ_a, ψ_b) by accounting for the expected chance agreement due to cluster size distribution M.
• Uses resampling (bootstrap) to compute instability across multiple perturbations of the data, averaging over pairs of objects to estimate expected disagreement.
• Derives a correction factor based on the expected Rand index under random clustering, which depends on cluster size distribution M, to normalize instability.
• Applies the normalized instability measure across a range of k values and selects the k with minimal instability as the estimated number of clusters.
• Employs both model-based (parametric bootstrap) and model-free (nonparametric bootstrap) resampling strategies to compute instability.
• Validates the method through extensive simulations across four synthetic clustering scenarios with varying k* and cluster shapes (circular vs. elongated).

Experimental results

Research questions

• RQ1 How does the distribution of cluster sizes M affect the performance of unnormalized cluster instability measures in k-selection?
• RQ2 Can a normalized instability measure that corrects for M improve k-estimation accuracy across the full range of k, especially for large k?
• RQ3 How do model-based and model-free approaches to instability computation compare in performance and convergence?
• RQ4 Does the proposed normalization allow instability-based methods to outperform established distance-based k-selection methods like the Gap statistic or Gaussian Mixture Model?
• RQ5 What is the convergence behavior of model-based and model-free instability estimates as the number of bootstrap samples B increases?

Key findings

• The normalized instability measure significantly outperforms unnormalized instability methods across all tested k values, particularly in scenarios with large k or uneven cluster sizes.
• In scenarios with k*=7 and elongated clusters, unnormalized methods failed entirely (performance dropped to zero), while normalized methods still correctly identified k* in a substantial proportion of cases.
• The Gaussian Mixture Model remained the top-performing distance-based method, but normalized instability methods outperformed all other distance-based methods, including the Gap and Jump statistics.
• Model-based and model-free instability approaches showed near-identical performance, with correlation coefficients between their instability paths ranging from 0.98 to 1.0.
• The two instability methods did not fully converge even at 5,000 bootstrap samples, suggesting potential divergence or slow convergence, though both stabilized in a narrow region around 0.038.
• The proposed normalization effectively controls for chance agreement due to cluster size, enabling reliable k-estimation even when cluster sizes are highly imbalanced.