[Paper Review] Medoid Silhouette clustering with automatic cluster number selection

Lars Lenssen, Erich Schubert|arXiv (Cornell University)|Sep 7, 2023

Advanced Clustering Algorithms Research4 citations

TL;DR

This paper proposes FastMSC and DynMSC, efficient algorithms for clustering via direct optimization of the Medoid Silhouette, achieving up to 10,464× speedup over the original PAMMEDSIL by leveraging caching and partial computation. It further introduces DynMSC, which automatically selects the optimal number of clusters by avoiding redundant computations across k values, enabling scalable and robust clustering on large datasets like MNIST with 30,000 samples.

ABSTRACT

The evaluation of clustering results is difficult, highly dependent on the evaluated data set and the perspective of the beholder. There are many different clustering quality measures, which try to provide a general measure to validate clustering results. A very popular measure is the Silhouette. We discuss the efficient medoid-based variant of the Silhouette, perform a theoretical analysis of its properties, provide two fast versions for the direct optimization, and discuss the use to choose the optimal number of clusters. We combine ideas from the original Silhouette with the well-known PAM algorithm and its latest improvements FasterPAM. One of the versions guarantees equal results to the original variant and provides a run speedup of $O(k^2)$. In experiments on real data with 30000 samples and $k$=100, we observed a 10464$ imes$ speedup compared to the original PAMMEDSIL algorithm. Additionally, we provide a variant to choose the optimal number of clusters directly.

Motivation & Objective

• To address the computational inefficiency of optimizing the Medoid Silhouette for clustering, especially for large datasets.
• To develop a fast, scalable algorithm that directly optimizes the Medoid Silhouette, avoiding the need for iterative search or heuristics.
• To enable automatic selection of the optimal number of clusters by reusing intermediate results across different k values.
• To provide a theoretically sound and empirically robust alternative to standard clustering evaluation and selection methods.

Proposed method

• Proposes FastMSC, a variant of PAM that caches nearest medoid distances to reduce redundant distance computations, achieving O(k²) speedup over PAMMEDSIL.
• Introduces a two-phase optimization: first compute medoid candidates using FasterPAM, then refine using Medoid Silhouette-based swaps.
• Develops DynMSC, a dynamic algorithm that incrementally builds clusterings from k=2 to k=max by reusing prior clustering structures and minimizing repeated distance calculations.
• Uses the Average Silhouette Width (ASW) as the objective function, with a novel formulation that allows efficient incremental updates during cluster refinement.
• Employs a partial computation strategy that avoids recalculating all pairwise distances when swapping medoids, significantly reducing time complexity.
• Integrates the Medoid Silhouette as both a clustering objective and a cluster number selection criterion, enabling end-to-end optimization.

Experimental results

Research questions

• RQ1 Can the Medoid Silhouette be efficiently optimized for large-scale clustering without relying on heuristics or multiple restarts?
• RQ2 How can the computational cost of evaluating the Medoid Silhouette be reduced to make it practical for datasets with tens of thousands of points?
• RQ3 Can the optimal number of clusters be selected automatically by reusing intermediate results across different k values?
• RQ4 Does direct optimization of the Medoid Silhouette yield better or more stable clusterings compared to standard k-medoids or k-means?
• RQ5 How does the performance of the proposed algorithms compare to existing implementations like scikit-learn’s k-means and FasterPAM?

Key findings

• FastMSC achieves a 10,464× speedup over the original PAMMEDSIL algorithm on a 30,000-sample dataset with k=100, demonstrating the effectiveness of caching and partial computation.
• DynMSC reduces the number of swaps by 2.15× compared to repeated FastMSC runs, resulting in a 1.97× speedup over repeated optimization for k=2 to 50.
• DynMSC is 9.01× faster than scikit-learn’s k-means with repeated Silhouette score computation, due to avoiding redundant distance calculations.
• The Average Medoid Silhouette satisfies desirable theoretical properties and correlates well with human-validated cluster quality in gene expression data.
• The method successfully identifies the optimal number of clusters by evaluating ASW across k values without recomputing all distances from scratch.
• On MNIST data, DynMSC achieves high performance with minimal redundant computation, outperforming both FastMSC and scikit-learn k-means in runtime and efficiency.