[Paper Review] Distributed Silhouette Algorithm: Evaluating Clustering on Big Data

Marco Gaido|arXiv (Cornell University)|Mar 24, 2023

Advanced Clustering Algorithms Research4 citations

TL;DR

This paper presents the first linear-complexity distributed algorithm for computing the Silhouette metric in big data environments, enabling efficient parallel evaluation of clustering quality. By pre-computing cluster-level statistics for squared Euclidean and cosine distances, the method reduces computational complexity from O(N²) to O(N), making it feasible for large-scale datasets in Apache Spark.

ABSTRACT

In the big data era, the key feature that each algorithm needs to have is the possibility of efficiently running in parallel in a distributed environment. The popular Silhouette metric to evaluate the quality of a clustering, unfortunately, does not have this property and has a quadratic computational complexity with respect to the size of the input dataset. For this reason, its execution has been hindered in big data scenarios, where clustering had to be evaluated otherwise. To fill this gap, in this paper we introduce the first algorithm that computes the Silhouette metric with linear complexity and can easily execute in parallel in a distributed environment. Its implementation is freely available in the Apache Spark ML library.

Motivation & Objective

• Address the scalability bottleneck of the Silhouette metric in big data environments, where its standard O(N²) complexity limits practical use.
• Overcome the inefficiency of existing implementations that require full pairwise distance computations across large datasets.
• Design a distributed algorithm that enables linear-time Silhouette computation while supporting parallel execution across cluster nodes.
• Implement and contribute the algorithm to Apache Spark ML library for production use in big data workloads.
• Ensure compatibility with common distance measures—specifically squared Euclidean and cosine distance—through optimized, pre-computed statistics.

Proposed method

• Pre-compute normalized feature vectors for each data point and cluster centroids to enable efficient distance aggregation.
• Define cluster-level summary vectors (e.g., ΩΓₖ) that store the sum of normalized features across all points in a cluster.
• Use these pre-computed vectors to compute average inter-cluster distances in O(D) time per point, avoiding pairwise comparisons.
• Apply the Silhouette formula sᵢ = (bᵢ − aᵢ)/max{aᵢ, bᵢ} using only pre-aggregated statistics, reducing complexity from O(N²D) to O(ND).
• Leverage Apache Spark’s distributed computing model to parallelize the computation across workers, with each worker processing a subset of data points.
• Support only specific distance measures (squared Euclidean and cosine) due to mathematical constraints—especially the non-linearity of the square root in standard Euclidean distance.

Experimental results

Research questions

• RQ1 Can the Silhouette metric be computed with linear time complexity in a distributed big data environment?
• RQ2 How can cluster-level statistics be pre-computed to avoid O(N²) pairwise distance calculations?
• RQ3 What distance measures allow for efficient aggregation of statistics to enable linear-time Silhouette computation?
• RQ4 What performance gains can be achieved over the standard O(N²) implementation in realistic big data workloads?
• RQ5 Is it feasible to integrate such an optimized algorithm into a widely used big data framework like Apache Spark?

Key findings

• The proposed algorithm reduces Silhouette computation complexity from O(N²D) to O(ND), achieving linear scalability with dataset size.
• On a single machine, the method achieved a maximum runtime of 23 seconds for a dataset of 150,000 points, while the standard implementation became infeasible beyond 100,000 points.
• The performance gap between the standard and proposed methods reached three orders of magnitude at 100,000 data points, as shown in logarithmic runtime plots.
• The algorithm is natively integrated into Apache Spark ML library under the Apache 2.0 license, enabling production use in distributed environments.
• The method is currently limited to squared Euclidean and cosine distances due to mathematical constraints—especially the non-linear square root in standard Euclidean distance, which prevents aggregation.
• The number of clusters significantly affects the computational cost, as the algorithm’s complexity scales with the number of clusters K.