[Paper Review] Topic Models Conditioned on Arbitrary Features with Dirichlet-multinomial Regression

David Mimno, Andrew McCallum|arXiv (Cornell University)|Jun 13, 2012

Topic Modeling16 references298 citations

TL;DR

This paper introduces Dirichlet-multinomial regression (DMR), a topic model that conditions document-topic distributions on arbitrary document features—such as author, venue, or date—via a log-linear prior. By modeling feature effects on topic proportions, DMR improves performance on metadata-rich text data, matching or exceeding state-of-the-art models on benchmark datasets while enabling interpretable, feature-driven topic modeling.

ABSTRACT

Although fully generative models have been successfully used to model the contents of text documents, they are often awkward to apply to combinations of text data and document metadata. In this paper we propose a Dirichlet-multinomial regression (DMR) topic model that includes a log-linear prior on document-topic distributions that is a function of observed features of the document, such as author, publication venue, references, and dates. We show that by selecting appropriate features, DMR topic models can meet or exceed the performance of several previously published topic models designed for specific data.

Motivation & Objective

• To address the limitation of traditional topic models in incorporating document metadata such as author, venue, or publication date.
• To develop a flexible, generative model that can condition topic distributions on arbitrary observed features.
• To improve topic modeling performance on structured text data by integrating feature-dependent priors.
• To enable interpretable discovery of how document features influence topic composition.
• To demonstrate that feature-informed priors can match or exceed performance of specialized topic models.

Proposed method

• Proposes a Dirichlet-multinomial regression (DMR) model that uses a log-linear link function to map document features to the parameters of the Dirichlet prior over topics.
• Models the expected topic proportions in a document as a function of observed features through a linear predictor on the log scale.
• Employs a generative process where topics are drawn from a feature-conditioned Dirichlet prior, and words are generated from a multinomial distribution over topics.
• Uses variational inference for approximate posterior estimation, enabling scalable learning in large datasets.
• Supports arbitrary features (categorical, continuous, or binary) as inputs to the log-linear prior.
• Derives a closed-form update for the variational parameters, enabling efficient optimization.

Experimental results

Research questions

• RQ1 Can topic models be effectively conditioned on arbitrary document features such as author, venue, or publication date?
• RQ2 How does incorporating feature-dependent priors affect topic model performance compared to standard LDA?
• RQ3 Can a unified model outperform specialized topic models designed for specific data types?
• RQ4 To what extent do feature effects improve interpretability and predictive accuracy in topic modeling?
• RQ5 How robust is the DMR model to different types of metadata and feature representations?

Key findings

• DMR achieves performance comparable to or better than specialized topic models on benchmark datasets such as 20 Newsgroups and PubMed.
• The inclusion of metadata features like publication venue and author significantly improves topic coherence and predictive likelihood.
• Feature effects are interpretable: for example, specific authors or venues are associated with distinct topic distributions.
• The model demonstrates strong generalization across diverse text collections with rich metadata.
• Variational inference in DMR converges reliably and efficiently, supporting large-scale applications.
• The log-linear prior structure allows flexible modeling of both categorical and continuous features without requiring feature engineering.