[Paper Review] Homogeneity of Cluster Ensembles

Brijnesh J. Jain|arXiv (Cornell University)|Feb 8, 2016

Advanced Clustering Algorithms Research23 references3 citations

TL;DR

This paper addresses the non-uniqueness of mean partitions in cluster ensembles, proposing homogeneity as a measure to assess how likely a unique mean exists. It establishes sufficient conditions for uniqueness, links homogeneity to cluster stability, and provides a computable lower bound to identify outlier partitions, showing that uniqueness is feasible in practice with real-world data via larger datasets or outlier removal.

ABSTRACT

The expectation and the mean of partitions generated by a cluster ensemble are not unique in general. This issue poses challenges in statistical inference and cluster stability. In this contribution, we state sufficient conditions for uniqueness of expectation and mean. The proposed conditions show that a unique mean is neither exceptional nor generic. To cope with this issue, we introduce homogeneity as a measure of how likely is a unique mean for a sample of partitions. We show that homogeneity is related to cluster stability. This result points to a possible conflict between cluster stability and diversity in consensus clustering. To assess homogeneity in a practical setting, we propose an efficient way to compute a lower bound of homogeneity. Empirical results using the k-means algorithm suggest that uniqueness of the mean partition is not exceptional for real-world data. Moreover, for samples of high homogeneity, uniqueness can be enforced by increasing the number of data points or by removing outlier partitions. In a broader context, this contribution can be placed as a further step towards a statistical theory of partitions.

Motivation & Objective

• Address the challenge of non-uniqueness in mean partitions within cluster ensembles, which undermines statistical inference, consistency, and stability.
• Investigate under what conditions the mean partition is unique, challenging the assumption that uniqueness is either exceptional or generic.
• Introduce homogeneity as a practical measure to quantify how close a sample of partitions is to having a unique mean.
• Explore the relationship between homogeneity and cluster stability, revealing a potential conflict between diversity in consensus clustering and stability in standard clustering.
• Provide a computable lower bound for homogeneity to guide practical selection of partitions that ensure uniqueness of the mean.

Proposed method

• Represent partitions as points in an orbit space endowed with an intrinsic metric derived from the Euclidean distance, enabling geometric analysis.
• Define Fréchet functions to formalize the mean partition as a minimizer of within-partition dissimilarity, using the intrinsic metric.
• Establish sufficient conditions for uniqueness of the mean partition, showing it holds when sample partitions lie within a sufficiently small ball in the orbit space.
• Introduce homogeneity as a lower-boundable measure of proximity to uniqueness, based on the spread of partitions in the orbit space.
• Propose a computable lower bound for homogeneity using the minimal distance between cluster assignments in the sample, enabling efficient outlier detection.
• Link homogeneity to cluster stability by showing that high homogeneity correlates with low instability, suggesting a trade-off between diversity and stability in consensus clustering.

Experimental results

Research questions

• RQ1 Under what conditions is the mean partition of a cluster ensemble uniquely defined?
• RQ2 Is uniqueness of the mean partition an exceptional or generic property in practical clustering scenarios?
• RQ3 How can homogeneity be formally defined and computed as a proxy for the likelihood of unique mean existence?
• RQ4 What is the relationship between homogeneity and cluster stability in consensus clustering?
• RQ5 Can homogeneity be leveraged to improve the reliability of consensus clustering by identifying and removing outlier partitions?

Key findings

• The mean partition is unique if all sample partitions lie within a sufficiently small ball in the orbit space, providing a geometric condition for uniqueness.
• Homogeneity is introduced as a measure of how close a sample is to having a unique mean, with higher homogeneity indicating greater likelihood of uniqueness.
• A computable lower bound for homogeneity is derived, which helps identify outlier partitions that prevent uniqueness and can be removed to enforce it.
• Empirical results on k-means applied to synthetic and UCI datasets show that uniqueness of the mean partition is not exceptional for real-world data.
• For samples with high homogeneity, uniqueness can be enforced by increasing the number of data points or removing outlier partitions, suggesting practical strategies for stable consensus clustering.
• Homogeneity is positively correlated with cluster stability, indicating a potential conflict between the goals of diversity in consensus clustering and stability in standard clustering.