[Paper Review] A flexible EM-like clustering algorithm for noisy data

Violeta Roizman, Matthieu Jonckheere+1 more|arXiv (Cornell University)|Jul 2, 2019

Bayesian Methods and Mixture Models8 citations

TL;DR

This paper proposes a robust, EM-like clustering algorithm that improves upon traditional Gaussian Mixture Model (GMM) clustering by incorporating a semi-parametric scale parameter estimation per data point, enabling effective handling of noise, outliers, and heavy-tailed distributions. The method maintains high performance across diverse data sets, including MNIST, NORB, and 20newsgroups, outperforming k-means, standard EM-GMM, and spectral clustering in both synthetic and real-world scenarios.

ABSTRACT

Though very popular, it is well known that the EM for GMM algorithm suffers from non-Gaussian distribution shapes, outliers and high-dimensionality. In this paper, we design a new robust clustering algorithm that can efficiently deal with noise and outliers in diverse data sets. As an EM-like algorithm, it is based on both estimations of clusters centers and covariances. In addition, using a semi-parametric paradigm, the method estimates an unknown scale parameter per data-point. This allows the algorithm to accommodate for heavier tails distributions and outliers without significantly loosing efficiency in various classical scenarios. We first derive and analyze the proposed algorithm in the context of elliptical distributions, showing in particular important insensitivity properties to the underlying data distributions. We then study the convergence and accuracy of the algorithm by considering first synthetic data. Then, we show that the proposed algorithm outperforms other classical unsupervised methods of the literature such as k-means, the EM for Gaussian mixture models and its recent modifications or spectral clustering when applied to real data sets as MNIST, NORB, and 20newsgroups.

Motivation & Objective

• To address the limitations of standard EM for GMM in handling non-Gaussian data, outliers, and high-dimensional noise.
• To develop a clustering algorithm that remains robust under heavy-tailed distributions and varying cluster scales without sacrificing efficiency.
• To improve clustering accuracy and stability in real-world data with significant noise and structural variability.
• To provide a flexible, EM-like framework that adapts to data-specific scale variations through per-point estimation.

Proposed method

• The algorithm estimates cluster centers and covariance matrices using an EM-like iterative procedure.
• It introduces a semi-parametric scale parameter per data point to model individual observation uncertainty and accommodate heavy-tailed distributions.
• The method assumes elliptical distributions and leverages Tyler’s theorem to ensure robust scatter matrix estimation.
• It uses a likelihood-based framework where the component weights and responsibilities are updated using a modified likelihood incorporating per-point scale parameters.
• The algorithm applies a robust estimation strategy that reduces sensitivity to outliers by downweighting extreme observations through adaptive scale estimation.
• Convergence is analyzed under elliptical distribution assumptions, with theoretical guarantees on consistency and asymptotic normality.

Experimental results

Research questions

• RQ1 How can an EM-like algorithm be made robust to outliers and non-Gaussian heavy-tailed distributions in clustering tasks?
• RQ2 What impact does per-data-point scale estimation have on clustering accuracy and robustness in high-dimensional or noisy data?
• RQ3 How does the proposed method compare to classical clustering algorithms like k-means, EM-GMM, and spectral clustering on real-world data?
• RQ4 To what extent does the algorithm maintain performance under varying cluster shapes, scales, and noise levels?
• RQ5 Can a semi-parametric approach improve robustness without significantly increasing computational complexity?

Key findings

• The proposed algorithm significantly outperforms k-means, EM-GMM, and spectral clustering on the MNIST, NORB, and 20newsgroups datasets in terms of clustering accuracy.
• The method demonstrates strong insensitivity to underlying data distribution shapes, particularly under heavy-tailed and non-Gaussian conditions.
• Robustness to outliers is achieved through per-point scale estimation, which effectively downweights extreme observations without distorting cluster structure.
• The algorithm maintains high performance even in high-dimensional settings where standard GMM-EM fails due to poor covariance estimation.
• Theoretical analysis confirms convergence and consistency under elliptical distributions, with asymptotic normality of the estimator established.
• Empirical results show that the method achieves better clustering quality than EM-GMM and its recent variants, especially in the presence of noise and outliers.