[Paper Review] Supervised Topic Models

David M. Blei, Jon McAuliffe|arXiv (Cornell University)|Mar 3, 2010

Bayesian Methods and Mixture ModelsComputer Science26 references1,316 citations

TL;DR

This paper introduces supervised latent Dirichlet allocation (sLDA), a probabilistic model that jointly models document words and response variables to improve predictive performance. By incorporating response variables into the topic modeling framework via variational inference and EM optimization, sLDA outperforms both unsupervised LDA followed by regression and lasso regression on real-world tasks like predicting movie ratings and U.S. Senate amendment tones.

ABSTRACT

We introduce supervised latent Dirichlet allocation (sLDA), a statistical model of labelled documents. The model accommodates a variety of response types. We derive an approximate maximum-likelihood procedure for parameter estimation, which relies on variational methods to handle intractable posterior expectations. Prediction problems motivate this research: we use the fitted model to predict response values for new documents. We test sLDA on two real-world problems: movie ratings predicted from reviews, and the political tone of amendments in the U.S. Senate based on the amendment text. We illustrate the benefits of sLDA versus modern regularized regression, as well as versus an unsupervised LDA analysis followed by a separate regression.

Motivation & Objective

• To develop a statistical model that jointly models document text and response variables for improved prediction in text analysis.
• To address the limitation of unsupervised LDA in predictive tasks, where topics may align with corpus structure (e.g., genre) rather than predictive features (e.g., sentiment).
• To provide a supervised alternative to unsupervised topic modeling that directly optimizes for predictive accuracy on labeled documents.
• To demonstrate that incorporating response variables into topic modeling yields better predictive performance than standard feature engineering with unsupervised topics or regularized regression.
• To extend the applicability of topic models beyond descriptive analysis to predictive modeling in real-world settings with diverse response types.

Proposed method

• Proposes supervised LDA (sLDA), a generative model where document topics are influenced by a response variable through a linear predictor in the topic proportion distribution.
• Uses a variational inference procedure to approximate the intractable posterior expectations required for maximum-likelihood estimation.
• Employs an EM algorithm to iteratively optimize model parameters, with E-step computing approximate posteriors and M-step updating topic and response parameters.
• Derives specific inference and estimation algorithms for exponential family response distributions, including Gaussian and Poisson responses.
• Applies the model to document-response pairs by modeling each document’s word distribution as a mixture of topics, where topic proportions are conditionally dependent on the response variable.
• Incorporates response-specific linear predictors into the Dirichlet prior of topic proportions, enabling topic structure to be guided by predictive relevance.

Experimental results

Research questions

• RQ1 Can a topic model be adapted to incorporate response variables to improve predictive performance over unsupervised LDA and standard regression?
• RQ2 How does sLDA compare to lasso regression when using raw word frequencies as features for prediction?
• RQ3 Does supervised topic modeling yield more interpretable and predictive topics than unsupervised LDA when the goal is prediction?
• RQ4 Can sLDA effectively model diverse response types, such as continuous ratings and categorical policy positions?
• RQ5 To what extent does incorporating response information into topic modeling reduce prediction error compared to standard feature engineering pipelines?

Key findings

• On movie review data, sLDA achieved a predictive R² of 0.432, slightly outperforming the best lasso model (0.426), demonstrating a 2% improvement in prediction accuracy.
• On the 109th U.S. Senate data, sLDA achieved a predictive R² of 0.27, representing an 80% improvement over the best lasso model (0.15).
• On the 110th U.S. Senate data, sLDA achieved a predictive R² of 0.23, a 43% improvement over the best lasso model (0.16).
• sLDA consistently outperformed the baseline of unsupervised LDA followed by linear regression across all datasets, showing superior predictive power.
• The model identified interpretable topics related to policy issues—such as right-wing health care amendments and left-wing grants/immigration in the 109th Senate—demonstrating practical interpretability.
• sLDA provides not only better predictions but also a latent topic structure that can be used for downstream analysis, unlike lasso which only yields a prediction rule.