[Paper Review] Superclustering by finding statistically significant separable groups of optimal gaussian clusters

O. I. Berngardt|arXiv (Cornell University)|Sep 5, 2023

Information Systems and Technology ApplicationsBusiness, Management and Accounting3 citations

TL;DR

This paper proposes a novel superclustering algorithm that first identifies the optimal number of Gaussian clusters via the Bayesian Information Criterion (BIC), then groups them into statistically separable superclusters using a Mahalanobis distance-based approach and DBSCAN with a dynamically tuned epsilon parameter. The method maximizes a matrix quality criterion to ensure statistically significant separability, achieving high Rand Index scores (up to 0.999) on synthetic datasets, and supports soft clustering and inference on new data.

ABSTRACT

The paper presents the algorithm for clustering a dataset by grouping the optimal, from the point of view of the BIC criterion, number of Gaussian clusters into the optimal, from the point of view of their statistical separability, superclusters. The algorithm consists of three stages: representation of the dataset as a mixture of Gaussian distributions - clusters, which number is determined based on the minimum of the BIC criterion; using the Mahalanobis distance, to estimate the distances between the clusters and cluster sizes; combining the resulting clusters into superclusters using the DBSCAN method by finding its hyperparameter (maximum distance) providing maximum value of introduced matrix quality criterion at maximum number of superclusters. The matrix quality criterion corresponds to the proportion of statistically significant separated superclusters among all found superclusters. The algorithm has only one hyperparameter - statistical significance level, and automatically detects optimal number and shape of superclusters based of statistical hypothesis testing approach. The algorithm demonstrates a good results on test datasets in noise and noiseless situations. An essential advantage of the algorithm is its ability to predict correct supercluster for new data based on already trained clusterer and perform soft (fuzzy) clustering. The disadvantages of the algorithm are: its low speed and stochastic nature of the final clustering. It requires a sufficiently large dataset for clustering, which is typical for many statistical methods.

Motivation & Objective

• To address the challenge of unsupervised clustering where no ground truth is available, by creating a method that approximates expert-labeled cluster structures through statistical significance.
• To develop a clustering framework that combines model-based (Gaussian mixture) and density-based (DBSCAN) approaches to detect meaningful, statistically separable superclusters.
• To enable prediction of cluster membership for new data and support soft (fuzzy) clustering, enhancing practical utility beyond hard partitioning.
• To automatically determine the optimal number and shape of superclusters using hypothesis testing, reducing reliance on arbitrary hyperparameters.

Proposed method

• The algorithm first models the data as a Gaussian Mixture Model (GMM) with the number of components selected to minimize the Bayesian Information Criterion (BIC).
• It computes Mahalanobis distances between GMM components to estimate statistical separation and cluster sizes, enabling robust inter-cluster distance measurement.
• It applies DBSCAN to the GMM components using a sequential search over the epsilon (maximum distance) hyperparameter, starting from small to large values.
• A novel matrix quality criterion (MC) is introduced, defined as the proportion of statistically significant superclusters among all detected superclusters, which is maximized to select the optimal epsilon.
• The algorithm uses statistical hypothesis testing at a user-defined significance level (α = 0.1) to assess the separability of each supercluster.
• The final output is a set of superclusters, each represented as a mixture of Gaussians, with soft clustering probabilities and support for inference on new data.

Figure 1: Clustering points into BIC-optimal mixture of Gaussian distributions. On the left is a separation by the number of clusters $N_{BIC}$ corresponding to the BIC minimum. The background corresponds to decision areas, all points of which the algorithm will refer to given cluster. Colors corres

Experimental results

Research questions

• RQ1 Can a clustering algorithm automatically detect the optimal number and shape of superclusters by combining model-based and density-based clustering with statistical significance testing?
• RQ2 How well can a superclustering method based on Gaussian mixture modeling and statistical separability outperform standard clustering algorithms on synthetic datasets with and without noise?
• RQ3 To what extent can such a method support soft clustering and generalization to new data, given only a trained model?
• RQ4 How does the choice of significance level α affect the stability and quality of the resulting superclustering?

Key findings

• The algorithm achieved a Rand Index (RI) of 0.999 on the 'Small blobs' noise-free dataset, indicating near-perfect agreement with expert labeling.
• On the '3 horseshoes' dataset, the algorithm achieved an RI of 0.987, demonstrating strong performance on complex, non-convex cluster shapes.
• In noisy conditions, such as the '2 snakes' dataset, the algorithm maintained an RI of 0.72–0.83, outperforming the agglomerative baseline (0.65–0.85) in the same range.
• The matrix quality criterion (MC) reached a maximum value of 1.0 in all tested noise-free cases, indicating that all detected superclusters were statistically significant.
• The method demonstrated the ability to perform soft clustering and predict cluster membership for new data using a trained model, a key advantage over many unsupervised methods.
• Despite strong performance, the algorithm is computationally slow and exhibits stochastic variability due to the EM initialization in GMM fitting, with results varying across runs on the same data.

Figure 2: A-D) Distributions of points within clusters (different colors correspond to different clusters) and distributions of Mahalanobis distances corresponding to them (orange and green correspond to distributions of intercluster distances $p_{ij},p_{ji}$ , red and blue correspond to cluster siz