[Paper Review] Semi-supervised NMF Models for Topic Modeling in Learning Tasks

Jamie Haddock, Lara Kassab+10 more|arXiv (Cornell University)|Oct 15, 2020

Text and Document Classification Technologies39 references4 citations

TL;DR

This paper proposes new semi-supervised nonnegative matrix factorization (SSNMF) models that integrate supervision into topic modeling via maximum likelihood estimation under specific uncertainty distributions. Using multiplicative update rules, the models achieve high classification accuracy—81.78% median on the 20 Newsgroups dataset—while learning coherent, interpretable topics that align with class labels.

ABSTRACT

We propose several new models for semi-supervised nonnegative matrix factorization (SSNMF) and provide motivation for SSNMF models as maximum likelihood estimators given specific distributions of uncertainty. We present multiplicative updates training methods for each new model, and demonstrate the application of these models to classification, although they are flexible to other supervised learning tasks. We illustrate the promise of these models and training methods on both synthetic and real data, and achieve high classification accuracy on the 20 Newsgroups dataset.

Motivation & Objective

• To develop SSNMF models that incorporate label information during dimensionality reduction to improve supervised learning performance.
• To motivate the proposed SSNMF models as maximum likelihood estimators under specific uncertainty distributions for data and labels.
• To design multiplicative update algorithms that support missing data and partial supervision.
• To evaluate the models on both synthetic and real-world data, particularly for text classification tasks.
• To demonstrate that SSNMF can yield topic representations that are both semantically coherent and effective for classification.

Proposed method

• Formulates SSNMF as a maximum likelihood estimation (MLE) problem under assumed distributions of uncertainty in data and labels.
• Introduces multiple SSNMF variants using different loss functions: I-divergence (KL divergence) and Frobenius norm.
• Derives multiplicative update rules for all proposed models to enable iterative optimization with nonnegativity constraints.
• Supports partial supervision and missing data through flexible objective function design and update rules.
• Applies the models to text data by decomposing document-term matrices into topic representations (dictionary) and document-topic weights (representation matrix).
• Uses the learned low-dimensional representation for downstream classification, with performance evaluated on benchmark datasets.

Experimental results

Research questions

• RQ1 Can semi-supervised NMF models be formally justified as maximum likelihood estimators under specific uncertainty assumptions?
• RQ2 How do different combinations of divergence measures (I-divergence and Frobenius norm) affect model performance in topic modeling and classification?
• RQ3 To what extent can SSNMF improve classification accuracy compared to standard NMF or baseline classifiers on text data?
• RQ4 How interpretable are the topics learned by SSNMF, and do they align with known class labels in real-world datasets?
• RQ5 What is the impact of partial supervision and missing data on the convergence and performance of SSNMF models?

Key findings

• The proposed SSNMF models, particularly the (D(·‖·), ‖·‖F)-SSNMF variant, achieved a median test classification accuracy of 81.78% on the 20 Newsgroups dataset.
• Topic modeling results showed that learned topics were semantically coherent and aligned with class labels—e.g., topics on Middle East, guns, and religion were correctly associated with the Politics and Religion classes.
• The (D(·‖·), ‖·‖F)-SSNMF model outperformed other SSNMF variants in classification accuracy, though not necessarily in objective function error minimization.
• The models successfully preserved label-relevant features during dimensionality reduction, avoiding suppression of important data structure.
• The learned topic representations enabled high-performance classification comparable to a higher-dimensional multinomial Naive Bayes baseline.
• Qualitative analysis confirmed that top keywords in each topic were meaningful and consistent with the corresponding class, indicating effective topic coherence.