[Paper Review] Flexible Mixture Modeling with the Polynomial Gaussian Cluster-Weighted Model

Antonio Punzo|arXiv (Cornell University)|Jul 4, 2012

Bayesian Methods and Mixture Models18 references7 citations

TL;DR

This paper introduces the polynomial Gaussian cluster-weighted model (CWM), a flexible finite mixture model that extends the linear Gaussian CWM by modeling nonlinear relationships between variables using polynomial regression within each component. It enables model-based clustering and classification with improved performance on both artificial and real data, particularly when component relationships are non-linear, outperforming standard mixtures of polynomial regressions and achieving higher Adjusted Rand Index (ARI) values.

ABSTRACT

In the mixture modeling frame, this paper presents the polynomial Gaussian cluster-weighted model (CWM). It extends the linear Gaussian CWM, for bivariate data, in a twofold way. Firstly, it allows for possible nonlinear dependencies in the mixture components by considering a polynomial regression. Secondly, it is not restricted to be used for model-based clustering only being contextualized in the most general model-based classification framework. Maximum likelihood parameter estimates are derived using the EM algorithm and model selection is carried out using the Bayesian information criterion (BIC) and the integrated completed likelihood (ICL). The paper also investigates the conditions under which the posterior probabilities of component-membership from a polynomial Gaussian CWM coincide with those of other well-established mixture-models which are related to it. With respect to these models, the polynomial Gaussian CWM has shown to give excellent clustering and classification results when applied to the artificial and real data considered in the paper.

Motivation & Objective

• To develop a flexible finite mixture model that captures nonlinear dependencies in bivariate data for improved clustering and classification.
• To extend the linear Gaussian CWM by incorporating polynomial regression in each mixture component to model non-linear conditional relationships.
• To provide a unified framework for model-based clustering and classification, not restricted to clustering alone.
• To demonstrate superior performance over existing models like finite mixtures of polynomial regressions on both synthetic and real-world data.
• To establish theoretical conditions under which the polynomial Gaussian CWM recovers posterior component memberships from other well-known mixture models.

Proposed method

• The model factorizes the joint density of (X,Y)' into the product of the marginal density of X and the conditional density of Y|x using polynomial regression in each component.
• Polynomial regression of degree d is used to model the functional relationship Y on X within each mixture component, allowing for flexible, non-linear structures.
• Maximum likelihood estimation is performed using the Expectation-Maximization (EM) algorithm to iteratively compute parameter estimates and posterior probabilities.
• Model selection is conducted using the Bayesian Information Criterion (BIC) and the Integrated Completed Likelihood (ICL), balancing fit and complexity.
• Theoretical derivation shows that under specific parameter constraints, the polynomial Gaussian CWM reduces to other established models such as finite mixtures of regressions.
• The model is applied to artificial and real data (e.g., 'places' dataset), with results evaluated via Adjusted Rand Index (ARI) and CW-plots.

Experimental results

Research questions

• RQ1 Can a polynomial Gaussian CWM effectively model nonlinear dependencies in mixture components where linear models fail?
• RQ2 How does the performance of the polynomial Gaussian CWM compare to finite mixtures of polynomial regressions in clustering and classification tasks?
• RQ3 Under what parameter constraints does the polynomial Gaussian CWM recover posterior component memberships from other well-known mixture models?
• RQ4 Does the polynomial Gaussian CWM achieve better clustering accuracy than existing elliptical mixture models on non-elliptical, non-Gaussian subpopulations?
• RQ5 How does the inclusion of labeled data affect classification performance in the polynomial Gaussian CWM framework?

Key findings

• The polynomial Gaussian CWM achieved an ARI of 0.208 on the 'places' real dataset using the EM algorithm, outperforming the finite mixture of quadratic regressions (ARI = 0.146).
• When fitted via the CEM algorithm, the ARI for the same dataset increased to 0.235, indicating improved classification stability and performance.
• The model successfully captured complex, non-linear relationships in bivariate data, as evidenced by the CW-plot showing clear component separation.
• Theoretical analysis confirmed that the polynomial Gaussian CWM can recover posterior component memberships from other models under specific parameter constraints.
• Model selection using BIC and ICL effectively guided the choice of optimal number of components and polynomial degree.
• Simulation results showed that incorporating labeled data (m ≤ 250) progressively improved classification accuracy on unlabeled observations, with ARI increasing on average as m increased.