[Paper Review] A Mixture of Matrix Variate Bilinear Factor Analyzers

Michael P. B. Gallaugher, Paul D. McNicholas|arXiv (Cornell University)|Dec 22, 2017

Bayesian Methods and Mixture Models27 references3 citations

TL;DR

This paper introduces a novel Mixture of Matrix Variate Bilinear Factor Analyzers (MMVBFA) model for clustering high-dimensional matrix variate data, combining matrix variate factor analysis with finite mixture modeling. Using an AECM algorithm for parameter estimation, the model achieves superior clustering performance on simulated and real datasets, including MNIST and Olivetti Faces, with BIC consistently selecting optimal components and factors.

ABSTRACT

Over the years data has become increasingly higher dimensional, which has prompted an increased need for dimension reduction techniques. This is perhaps especially true for clustering (unsupervised classification) as well as semi-supervised and supervised classification. Although dimension reduction in the area of clustering for multivariate data has been quite thoroughly discussed within the literature, there is relatively little work in the area of three-way, or matrix variate, data. Herein, we develop a mixture of matrix variate bilinear factor analyzers (MMVBFA) model for use in clustering high-dimensional matrix variate data. This work can be considered both the first matrix variate bilinear factor analysis model as well as the first MMVBFA model. Parameter estimation is discussed, and the MMVBFA model is illustrated using simulated and real data.

Motivation & Objective

• Address the lack of dimension reduction techniques for model-based clustering of three-way (matrix variate) data.
• Develop a flexible, scalable method to handle high-dimensional matrix data in clustering applications.
• Extend factor analysis to matrix variate data by incorporating bilinear structures and mixture modeling.
• Enable simultaneous clustering and dimension reduction for matrix variate data, improving on existing multivariate and matrix variate approaches.
• Provide a framework that supports extensions to non-normal distributions, such as t and skewed matrix variate factors.

Proposed method

• Propose a Mixture of Matrix Variate Bilinear Factor Analyzers (MMVBFA) model as a generalization of bilinear PCA.
• Model each component using a matrix variate normal distribution with bilinear factor structure: $\mathbf{X}_{g} = \mathbf{M}_{g} + \mathbf{A}_{g}\mathbf{Z}_{g}\mathbf{B}_{g}^{\top} + \mathbf{E}_{g}$, where $\mathbf{Z}_{g}$ is a low-dimensional latent factor.
• Use an alternating expectation conditional maximization (AECM) algorithm for parameter estimation.
• Incorporate component-specific mean matrices $\mathbf{M}_{g}$, row and column factor loading matrices $\mathbf{A}_{g}, \mathbf{B}_{g}$, and component mixing proportions $\pi_{g}$.
• Apply the Bayesian Information Criterion (BIC) to select the optimal number of components and latent factors.
• Leverage the matrix variate normal assumption to allow for future extensions to matrix variate t, skew-elliptical, and generalized hyperbolic factor analyzers.

Experimental results

Research questions

• RQ1 Can a bilinear factor model be effectively extended to matrix variate data for clustering purposes?
• RQ2 How well does the MMVBFA model perform in clustering high-dimensional matrix variate data compared to existing methods?
• RQ3 Can the model automatically select the optimal number of components and latent factors using BIC?
• RQ4 To what extent does semi-supervised learning improve clustering performance on real-world image data?
• RQ5 Can the model detect meaningful subgroups in matrix variate data, such as facial features like glasses?

Key findings

• The MMVBFA model achieved perfect classification in all simulated datasets, with BIC correctly selecting the true number of components and factors.
• On the MNIST dataset, the model achieved an average ARI of 0.93 and MCR of 0.018 at 75% supervision, outperforming prior work by Gallaugher and McNicholas (2018a).
• For the MNIST data, the number of row and column factors selected by BIC was most frequently between 13 and 15 across all supervision levels.
• In the Olivetti Faces dataset, BIC selected three components with 26 row and 23 column factors, and component 3 was clearly associated with the presence of glasses.
• All faces with glasses were classified into component 3, except two classified into component 2, demonstrating the model’s ability to detect meaningful facial features.
• Heatmaps of estimated location matrices showed improved image quality with increased supervision, and the model maintained consistent performance across 25 runs, indicating robustness.