[Paper Review] Hybridization of Expectation-Maximization and K-Means Algorithms for Better Clustering Performance

D. Raja Kishor, N. Venkateswarlu|arXiv (Cornell University)|Mar 25, 2016

Bayesian Methods and Mixture Models8 references3 citations

TL;DR

This paper proposes a hybrid clustering algorithm combining K-Means and Expectation-Maximization (EM) to accelerate convergence while maintaining high clustering quality. By leveraging K-Means for fast initial centroid estimation and EM for refined parameter optimization, the hybrid approach achieves significantly reduced execution time—consistently outperforming standard EM in speed, with comparable or better clustering fitness and lower Sum of Squared Errors (SSE) across six datasets, including synthetic ones.

ABSTRACT

The present work proposes hybridization of Expectation-Maximization (EM) and K-Means techniques as an attempt to speed-up the clustering process. Though both K-Means and EM techniques look into different areas, K-means can be viewed as an approximate way to obtain maximum likelihood estimates for the means. Along with the proposed algorithm for hybridization, the present work also experiments with the Standard EM algorithm. Six different datasets are used for the experiments of which three are synthetic datasets. Clustering fitness and Sum of Squared Errors (SSE) are computed for measuring the clustering performance. In all the experiments it is observed that the proposed algorithm for hybridization of EM and K-Means techniques is consistently taking less execution time with acceptable Clustering Fitness value and less SSE than the standard EM algorithm. It is also observed that the proposed algorithm is producing better clustering results than the Cluster package of Purdue University.

Motivation & Objective

• To address the slow convergence of the EM algorithm in clustering by integrating faster K-Means initialization.
• To improve computational efficiency without sacrificing clustering quality.
• To develop a hybrid framework that leverages the strengths of both K-Means and EM for better overall performance.
• To evaluate the proposed method against standard EM and a well-known clustering package (Purdue University's Cluster package).

Proposed method

• The hybrid algorithm begins with K-Means to compute initial cluster centroids and assignments.
• These initial estimates are then used as starting points for the EM algorithm to refine cluster parameters via maximum likelihood estimation.
• The EM step iteratively updates cluster means, covariances, and mixing weights using the E-step (responsibility calculation) and M-step (parameter re-estimation).
• The algorithm alternates between K-Means initialization and EM refinement until convergence is reached.
• Clustering performance is evaluated using Clustering Fitness and Sum of Squared Errors (SSE).
• The method is tested on six datasets, including three synthetic ones, to ensure robustness across data types.

Experimental results

Research questions

• RQ1 Can combining K-Means and EM reduce execution time while preserving or improving clustering accuracy?
• RQ2 How does the hybrid approach compare to standard EM in terms of convergence speed and clustering quality?
• RQ3 Does the hybrid method outperform existing clustering packages, such as Purdue University’s Cluster package?
• RQ4 To what extent does K-Means initialization improve the convergence behavior of EM in clustering tasks?

Key findings

• The hybrid algorithm consistently achieved lower execution time than the standard EM algorithm across all six datasets.
• The proposed method produced lower Sum of Squared Errors (SSE) than standard EM, indicating better clustering compactness.
• Clustering Fitness values were acceptable and competitive, showing that the hybrid approach maintains high-quality clustering.
• The hybrid algorithm outperformed the Purdue University Cluster package in both execution time and clustering quality metrics.
• The combination of K-Means initialization and EM refinement led to faster convergence and improved stability in clustering results.