[Paper Review] On the EM-Tau algorithm: a new EM-style algorithm with partial E-steps

Val Andrei Fajardo, Jiaxi Liang|arXiv (Cornell University)|Nov 21, 2017

Blind Source Separation Techniques16 references3 citations

TL;DR

This paper introduces the EM-Tau algorithm, a novel EM-style method that accelerates convergence by performing partial E-steps using a dynamic thresholding policy based on membership probability heaps. By selectively re-evaluating only the most uncertain data points (those in the top τ% of membership probability heaps), EM-Tau achieves up to 60% faster runtime than traditional EM with less than 0.2% increase in classification error, maintaining high accuracy while drastically reducing computational cost on large datasets.

ABSTRACT

The EM algorithm is one of many important tools in the field of statistics. While often used for imputing missing data, its widespread applications include other common statistical tasks, such as clustering. In clustering, the EM algorithm assumes a parametric distribution for the clusters, whose parameters are estimated through a novel iterative procedure that is based on the theory of maximum likelihood. However, one major drawback of the EM algorithm, that renders it impractical especially when working with large datasets, is that it often requires several passes of the data before convergence. In this paper, we introduce a new EM-style algorithm that implements a novel policy for performing partial E-steps. We call the new algorithm the EM-Tau algorithm, which can approximate the traditional EM algorithm with high accuracy but with only a fraction of the running time.

Motivation & Objective

• To address the slow convergence of the traditional EM algorithm on large datasets, which requires multiple full passes over the data.
• To develop a computationally efficient alternative to EM that maintains high accuracy by reducing E-step computation.
• To introduce a novel partial E-step policy based on dynamic thresholding of membership probability heaps.
• To evaluate the trade-off between computational efficiency and approximation accuracy in clustering tasks.

Proposed method

• The EM-Tau algorithm performs a partial E-step by identifying and re-evaluating only the top τ% of data points with the highest membership probability heaps across clusters.
• It maintains a heap structure for each cluster to track the most uncertain data points, defined as those with high posterior probability values.
• The algorithm dynamically updates the set of active points at each iteration, focusing computation only on points that are likely to change membership.
• The M-step proceeds as in the standard EM algorithm, updating parameters using only the active points from the partial E-step.
• The threshold τ controls the trade-off between speed and accuracy, with higher τ values approaching full EM performance.
• The method is theoretically grounded in the convergence properties of EM, with the authors proving that the partial E-step policy preserves monotonic likelihood increase under mild conditions.

Experimental results

Research questions

• RQ1 Can a partial E-step policy based on dynamic thresholding of membership probabilities significantly reduce EM computation time without sacrificing clustering accuracy?
• RQ2 How does the choice of τ, the threshold parameter, affect the trade-off between runtime and approximation quality in EM-Tau?
• RQ3 Does EM-Tau maintain the monotonic likelihood increase property of the standard EM algorithm despite using partial E-steps?
• RQ4 How does EM-Tau compare to existing EM variants like EM*, Sparse EM, and Lazy EM in terms of speed and accuracy on real-world clustering tasks?
• RQ5 Can the EM-Tau framework be extended with additional acceleration techniques such as revisiting mechanisms or partial M-steps?

Key findings

• With τ = 20, EM-Tau achieved a 60% speedup over traditional EM, reducing runtime from 287.2 seconds to 117.4 seconds, while increasing classification error by only 0.0007.
• At τ = 25, EM-Tau achieved a classification error of 0.057227, compared to 0.057394 for traditional EM, indicating near-identical accuracy with significantly reduced computation.
• The EM-Tau(25) variant required only 56 iterations, compared to 71 for traditional EM, demonstrating faster convergence despite partial E-steps.
• The EM* algorithm failed to produce reliable results in the experiments, likely due to overly aggressive pruning of active points, highlighting the advantage of EM-Tau’s more balanced thresholding policy.
• K-means clustering was significantly less accurate (classification error 0.147184) than both EM and EM-Tau, underscoring the value of probabilistic modeling.
• The confusion matrices show that EM-Tau(25) closely approximates the performance of traditional EM, with minimal misclassification across all clusters.