[Paper Review] Automatic Relevance Determination in Nonnegative Matrix Factorization with the \beta-Divergence

Vincent Y. F. Tan, Cédric Févotte|arXiv (Cornell University)|Nov 25, 2011

Face and Expression Recognition4 citations

TL;DR

This paper proposes a Bayesian nonnegative matrix factorization (NMF) framework using automatic relevance determination (ARD) with the β-divergence to automatically estimate the model order K. By tying the scale parameters of the dictionary and activation matrices, the method induces group sparsity, pruning irrelevant components via MM algorithms that monotonically minimize the objective, achieving superior performance on synthetic, music, and stock data compared to baseline methods.

ABSTRACT

This paper addresses the estimation of the latent dimensionality in nonnegative matrix factorization (NMF) with the \beta-divergence. The \beta-divergence is a family of cost functions that includes the squared Euclidean distance, Kullback-Leibler and Itakura-Saito divergences as special cases. Learning the model order is important as it is necessary to strike the right balance between data fidelity and overfitting. We propose a Bayesian model based on automatic relevance determination in which the columns of the dictionary matrix and the rows of the activation matrix are tied together through a common scale parameter in their prior. A family of majorization-minimization algorithms is proposed for maximum a posteriori (MAP) estimation. A subset of scale parameters is driven to a small lower bound in the course of inference, with the effect of pruning the corresponding spurious components. We demonstrate the efficacy and robustness of our algorithms by performing extensive experiments on synthetic data, the swimmer dataset, a music decomposition example and a stock price prediction task.

Motivation & Objective

• To address the critical challenge of selecting the correct model order K in nonnegative matrix factorization (NMF), balancing data fidelity and overfitting.
• To develop a Bayesian framework that enables automatic relevance determination (ARD) for NMF using the β-divergence family of cost functions.
• To design computationally efficient, monotonic majorization-minimization (MM) algorithms that estimate W, H, and K simultaneously.
• To improve robustness and performance on real-world data, including audio, images, and financial time series, by incorporating flexible noise models via β.
• To provide a principled, unified approach for model order selection that outperforms existing heuristic or sampling-based methods.

Proposed method

• Formulates a Bayesian NMF model where the columns of W and rows of H are linked via a common scale parameter in the prior, enabling ARD.
• Uses the β-divergence as a flexible cost function, subsuming Euclidean, Kullback-Leibler, and Itakura-Saito divergences.
• Derives a family of majorization-minimization (MM) algorithms that ensure monotonic decrease of the objective function at each iteration.
• Employs auxiliary functions to construct local upper bounds of the objective, leading to multiplicative update rules with linear complexity in F, K, and N.
• Applies group-sparsity regularization through the scale parameters, driving irrelevant components toward zero during inference.
• Introduces a method of moments for adaptive hyperparameter selection, particularly for the dispersion parameter φ.

Experimental results

Research questions

• RQ1 Can automatic relevance determination (ARD) be effectively applied to NMF with the β-divergence to enable automatic model order selection?
• RQ2 How does the proposed ARD-NMF framework perform in comparison to standard NMF and other model order selection techniques on diverse data types?
• RQ3 To what extent does the flexibility of the β-divergence improve factorization quality and robustness across different noise models?
• RQ4 Can the MM-based optimization framework guarantee monotonic convergence while maintaining computational efficiency?
• RQ5 How well does the method handle missing data, as demonstrated in the stock price prediction task?

Key findings

• The ℓ2-ARD method achieved an NKLD of approximately 0.2 on stock price prediction, significantly outperforming standard KL-NMF, which had an NKLD above 0.23 across all K values.
• For the same task, ℓ1-ARD achieved an NKLD of about 0.23 when K_eff = 5, outperforming KL-NMF (NKLD ≈ 0.25), even with fewer components retained.
• The method of moments enabled effective, data-dependent selection of hyperparameters, with a recommended small value of a relative to F + N.
• On the swimmer dataset and music decomposition, the proposed algorithms recovered the correct model order and produced higher-quality decompositions than baseline methods.
• The ℓ2-ARD method was robust across a wide range of β and φ values, with β = 0.5 and φ = 10 yielding the best performance on prediction tasks.
• The method outperformed a missing-data-adapted version of Mørup & Hansen’s approach, which yielded a mean NKLD of 0.37 ± 0.03, indicating poor data fit.