[Paper Review] Visualizing Topics with Multi-Word Expressions

David M. Blei, John Lafferty|ArXiv.org|Jul 6, 2009

Advanced Text Analysis Techniques24 references90 citations

TL;DR

This paper proposes 'turbo topics,' a method to enhance topic visualization in LDA models by identifying significant multi-word expressions (n-grams) through recursive permutation testing. By leveraging topic-annotated corpora and a back-off language model, the approach improves interpretability by revealing context-rich phrases—such as 'phase diagram' or 'supreme court'—that better convey topic meaning than unigram lists alone.

ABSTRACT

We describe a new method for visualizing topics, the distributions over terms that are automatically extracted from large text corpora using latent variable models. Our method finds significant $n$-grams related to a topic, which are then used to help understand and interpret the underlying distribution. Compared with the usual visualization, which simply lists the most probable topical terms, the multi-word expressions provide a better intuitive impression for what a topic is "about." Our approach is based on a language model of arbitrary length expressions, for which we develop a new methodology based on nested permutation tests to find significant phrases. We show that this method outperforms the more standard use of $χ^2$ and likelihood ratio tests. We illustrate the topic presentations on corpora of scientific abstracts and news articles.

Motivation & Objective

• Improve the interpretability of topic models by moving beyond unigram term lists to include meaningful multi-word expressions.
• Address the limitation of standard topic visualization, where single terms lack contextual coherence and thematic clarity.
• Develop a statistically robust method to identify significant n-grams that are specifically relevant to each topic, preserving the simplicity of unigram topic models.
• Enable more intuitive and accurate understanding of topics in large text corpora, such as scientific abstracts and news articles.
• Provide a generalizable framework applicable to any topic model with word-level topic assignments, not limited to LDA.

Proposed method

• First, fit a standard LDA model to the corpus and assign the most probable topic to each word in the document using posterior inference.
• Construct a topic-annotated corpus where each word is labeled with its inferred topic, enabling context-aware co-occurrence analysis.
• Apply a recursive, back-off language model to model n-grams of arbitrary length, allowing for variable-length phrase discovery.
• Use a distribution-free nested permutation test to assess the statistical significance of n-grams, avoiding reliance on asymptotic approximations.
• Iteratively expand phrases by testing co-occurrence significance in topical contexts, stopping when no further significant n-grams are found.
• Combine significant n-grams with unigram probabilities, adjusting for subsumption (e.g., merging 'New York Mets' with 'New York' if nested), to produce a unified, interpretable visualization.

Experimental results

Research questions

• RQ1 Can multi-word expressions provide a more intuitive and accurate representation of topic content than unigram term lists?
• RQ2 How can significant n-grams be reliably detected in topic-specific contexts without relying on asymptotic test statistics?
• RQ3 Does a recursive, permutation-based testing procedure outperform traditional chi-squared or likelihood ratio tests in small-sample, topical settings?
• RQ4 To what extent do the resulting turbo topics improve interpretability in real-world corpora such as news articles and scientific abstracts?
• RQ5 Can this method be generalized to other topic models beyond LDA, provided word-topic assignments are available?

Key findings

• The permutation test-based method for identifying significant n-grams outperforms standard chi-squared and likelihood ratio tests in small-sample settings typical of topic-specific phrase discovery.
• Turbo topics significantly improve topic interpretability: for example, 'indiana jones' and 'sex in the city' clarify ambiguous unigrams like 'jones' and 'city' in news topics.
• In physics abstracts, phrases such as 'black hole mass' and 'supermassive black holes' provide clearer thematic context than isolated terms like 'black' or 'holes'.
• The method successfully identifies contextually meaningful phrases such as 'the california supreme court', which refines the interpretation of general terms like 'court' and 'supreme'.
• The recursive, back-off language model enables detection of multi-word expressions of varying lengths within a coherent statistical framework, enhancing phrase discovery accuracy.
• The approach maintains computational efficiency and statistical simplicity of LDA while adding interpretative power through context-aware phrase extraction.