[Paper Review] Multi-Node EM Algorithm for Finite Mixture Models

Sharon Lee, Geoffrey J. McLachlan+1 more|arXiv (Cornell University)|May 14, 2020

Algorithms and Data Compression23 references4 citations

TL;DR

This paper proposes a multi-node EM algorithm for finite mixture models that enables efficient parallelization across single machines and small multi-node clusters. By distributing the E-step across observations and components and the M-step across mixture components, the method achieves significant speedups—up to 79% reduction in computation time with 18 threads—demonstrating superior performance in R compared to MATLAB for large-scale mixture model fitting.

ABSTRACT

Finite mixture models are powerful tools for modelling and analyzing heterogeneous data. Parameter estimation is typically carried out using maximum likelihood estimation via the Expectation-Maximization (EM) algorithm. Recently, the adoption of flexible distributions as component densities has become increasingly popular. Often, the EM algorithm for these models involves complicated expressions that are time-consuming to evaluate numerically. In this paper, we describe a parallel implementation of the EM-algorithm suitable for both single-threaded and multi-threaded processors and for both single machine and multiple-node systems. Numerical experiments are performed to demonstrate the potential performance gain n different settings. Comparison is also made across two commonly used platforms - R and MATLAB. For illustration, a fairly general mixture model is used in the comparison.

Motivation & Objective

• To address the computational bottleneck in fitting finite mixture models with flexible, non-normal component densities such as skew-t distributions.
• To develop a scalable, parallel EM algorithm suitable for both single-machine and small multi-node computing environments.
• To compare the performance of the multi-node EM algorithm across R and MATLAB platforms in terms of speed and stability.
• To evaluate the impact of thread count and networked architectures on runtime efficiency for computationally intensive mixture models.

Proposed method

• The E-step is parallelized by assigning one thread per observation-component pair, enabling independent computation of posterior probabilities.
• The M-step is parallelized by distributing the parameter update computations across g threads, one per mixture component.
• The algorithm is implemented in both R and MATLAB to enable cross-platform performance benchmarking on identical hardware and data.
• The method supports both single-core and multi-core systems, as well as small local networks of machines, using message-passing for inter-node communication.
• A finite mixture of canonical fundamental skew-t (CFUST) distributions is used as the target model for numerical evaluation.
• Performance is evaluated using the Australian Institute of Sport (AIS) dataset across 100 trials with varying thread counts (m=1 to 20).

Experimental results

Research questions

• RQ1 How does the multi-node EM algorithm improve computation time for finite mixture models with complex component densities?
• RQ2 What is the relative performance of R versus MATLAB in implementing the parallel EM algorithm across different thread counts?
• RQ3 How does the number of threads affect the speedup, and at what point do diminishing returns or overheads emerge?
• RQ4 How effective is the algorithm in a multi-node cluster setting, and does network communication overhead limit scalability?

Key findings

• The R implementation achieved a maximum time reduction of 76.97% with 11 threads and 76.44% with 12 threads, indicating strong performance gains even at moderate thread counts.
• MATLAB showed a slower initial performance than R but overtook it at m ≥ 15, achieving a peak time reduction of 80.41% with 20 threads.
• The standard deviation of computation time was consistently lower in MATLAB than in R, indicating more stable performance across trials, especially at higher thread counts.
• For the multi-node setting with three machines, only a 4.5% average reduction in time was observed, suggesting that network overhead limits gains for small datasets.
• The performance trend in R showed a slight increase in computation time at m=12, likely due to thread management overhead, while MATLAB maintained a steady decline in time with increasing threads.
• The multi-node EM algorithm demonstrated significant scalability on single machines, with both platforms achieving over 75% time reduction at moderate thread counts, confirming its utility for large-scale mixture modeling.