[Paper Review] Multivariate normal mixture modeling, clustering and classification with the rebmix package

Marko Nagode|arXiv (Cornell University)|Jan 26, 2018

Bayesian Methods and Mixture Models10 references3 citations

TL;DR

This paper introduces the rebmix R package for multivariate normal mixture modeling, clustering, and classification using the REBMIX algorithm, which estimates component parameters and weights individually rather than simultaneously. It demonstrates superior computational speed and numerical stability for large datasets, especially with histogram preprocessing, achieving 93.38% clustering accuracy and 6.62% classification error on simulated data with unrestricted covariance matrices.

ABSTRACT

The rebmix package provides R functions for random univariate and multivariate finite mixture model generation, estimation, clustering and classification. The paper is focused on multivariate normal mixture models with unrestricted variance-covariance matrices. The objective is to show how to generate datasets for a known number of components, numbers of observations and component parameters, how to estimate the number of components, component weights and component parameters and how to predict cluster and class membership based upon a model trained by the REBMIX algorithm. The accompanying plotting, bootstrapping and other features of the package are dealt with, too. For demonstration purpose a multivariate normal dataset with unrestricted variance-covariance matrices is studied.

Motivation & Objective

• To develop and present the rebmix R package for finite mixture modeling, clustering, and classification with multivariate normal components.
• To demonstrate the REBMIX algorithm’s ability to estimate component parameters and weights individually, improving numerical stability and computational speed.
• To evaluate the package’s performance on large datasets using histogram preprocessing, comparing it to EM-based methods.
• To provide tools for generating synthetic datasets, estimating component counts, and classifying observations with uncertainty measures.
• To enable researchers to use rebmix as a robust alternative for initial parameter estimation in mixture modeling.

Proposed method

• The REBMIX algorithm estimates component weights and parameters (mean vector and covariance matrix) individually using maximum likelihood principles, avoiding simultaneous estimation used in EM.
• The package supports multivariate normal mixtures with unrestricted variance-covariance matrices, using predictive density estimation and component-specific parameter fitting.
• Preprocessing via histogram, Parzen window, or k-nearest neighbors is applied to improve estimation stability, especially for large datasets.
• Information criteria (e.g., BIC, AIC, ICL) are used to select the optimal number of components, with model selection based on minimizing the criterion.
• Clustering is performed using the REBMIX algorithm, and classification is conducted via the RCLSMIX method using trained models on training data.
• Bootstrapping and entropy-based evaluation (entropy and entropy decrease) are used to assess clustering quality and stability.

Experimental results

Research questions

• RQ1 How does the REBMIX algorithm compare to EM-based methods in terms of computational speed and numerical stability for large multivariate normal mixture datasets?
• RQ2 What is the optimal preprocessing method (histogram, Parzen window, k-NN) for improving estimation accuracy and speed in finite mixture modeling?
• RQ3 Can the rebmix package reliably estimate the number of components, component weights, and parameters in multivariate normal mixtures with unrestricted covariance matrices?
• RQ4 How accurate is the classification performance of the RCLSMIX method when applied to test data from simulated multivariate normal mixtures?
• RQ5 To what extent does the REBMIX algorithm improve clustering quality, as measured by entropy and entropy decrease?

Key findings

• The rebmix package achieved a clustering accuracy of 93.38% on a simulated multivariate normal dataset with 2000 observations and 5 components, with 77.9% probability of correct cluster assignment for the optimal component count.
• The classification error rate was 6.62% on the test set, indicating strong predictive performance for multiclass classification using the RCLSMIX method.
• The REBMIX algorithm demonstrated superior computational speed and numerical stability on large datasets, especially when using histogram preprocessing.
• The entropy decrease metric showed a steady increase from 2.39 to 3150.78 as the number of clusters decreased from 2 to 1, indicating effective clustering structure detection.
• The optimal number of components was identified as 5 using the BIC criterion, with the lowest BIC value achieved at this configuration.
• The package outperformed EM-based alternatives in speed and stability for large datasets, though slightly less accurate on small datasets due to preprocessing effects.