[Paper Review] Graph-Sparse LDA: A Topic Model with Structured Sparsity

Finale Doshi‐Velez, Byron Wallace+1 more|arXiv (Cornell University)|Oct 16, 2014

Topic Modeling17 references4 citations

TL;DR

Graph-Sparse LDA is a Bayesian nonparametric topic model that leverages structured vocabularies—such as medical ontologies or gene pathways—to induce sparse, interpretable topics by modeling topics as distributions over concept-words within a directed acyclic graph (DAG). By using graph-structured relationships to guide sparsity, the model achieves state-of-the-art prediction performance while producing significantly more interpretable topic summaries than standard sparse topic models.

ABSTRACT

Originally designed to model text, topic modeling has become a powerful tool for uncovering latent structure in domains including medicine, finance, and vision. The goals for the model vary depending on the application: in some cases, the discovered topics may be used for prediction or some other downstream task. In other cases, the content of the topic itself may be of intrinsic scientific interest. Unfortunately, even using modern sparse techniques, the discovered topics are often difficult to interpret due to the high dimensionality of the underlying space. To improve topic interpretability, we introduce Graph-Sparse LDA, a hierarchical topic model that leverages knowledge of relationships between words (e.g., as encoded by an ontology). In our model, topics are summarized by a few latent concept-words from the underlying graph that explain the observed words. Graph-Sparse LDA recovers sparse, interpretable summaries on two real-world biomedical datasets while matching state-of-the-art prediction performance.

Motivation & Objective

• To address the challenge of low interpretability in high-dimensional topic models, especially in scientific domains with structured vocabularies.
• To leverage existing controlled vocabularies—such as ICD9-CM, MeSH, or gene pathways—as prior knowledge to guide topic discovery.
• To develop a topic model that produces sparse, concept-based topic summaries that are meaningful to domain experts.
• To maintain or exceed the predictive performance of state-of-the-art sparse topic models while enhancing interpretability.
• To design an efficient inference procedure that exploits graph structure to identify sparse, high-likelihood solutions.

Proposed method

• Introduces a hierarchical topic model where topics are distributions over concept-words in a DAG-structured vocabulary, rather than over raw words.
• Uses a noise process informed by the graph structure to generate observed words from concept-words, allowing a concept to explain its ancestors and descendants.
• Employs a novel Gibbs sampling inference procedure that favors moves along manifolds of constant likelihood to efficiently explore sparse solutions.
• Applies a Bayesian nonparametric prior (e.g., Chinese Restaurant Process) to allow automatic discovery of the number of topics.
• Incorporates the graph structure via a sparsity-inducing matrix P that defines which observed words can be generated by which concept-words.
• Uses the graph to constrain the support of topic distributions, ensuring that only concept-words with meaningful semantic relationships contribute to topics.

Experimental results

Research questions

• RQ1 Can structured vocabularies such as medical ontologies be used to improve the interpretability of topic models without sacrificing predictive performance?
• RQ2 How can graph-structured relationships between words be leveraged to induce sparsity in topic models in a way that preserves data explanation capacity?
• RQ3 Does incorporating domain-specific knowledge via a DAG improve the clinical or scientific meaningfulness of discovered topics compared to standard sparse topic models?
• RQ4 Can the model efficiently discover both the number of topics and their sparse, concept-based representations using only the graph structure and observed data?
• RQ5 Is the proposed inference procedure more effective at finding sparse, interpretable solutions than standard Gibbs sampling in sparse LDA?

Key findings

• Graph-Sparse LDA achieved state-of-the-art or better prediction performance on two real-world biomedical datasets: autism patient diagnoses and MeSH-annotated biomedical abstracts.
• The model produced significantly sparser topic descriptions—using only a few key concept-words—while maintaining or improving data explanation capacity compared to Latent IBP Compound Dirichlet Allocation.
• On the autism dataset, Graph-Sparse LDA recovered clinically relevant subtypes using only a few high-level diagnostic concepts (e.g., 'Epilepsy' summarizing specific seizure types).
• On the MeSH-annotated abstracts, the model identified concise, meaningful groupings of MeSH terms that aligned with known biomedical topics and improved interpretability.
• The inference procedure efficiently identified sparse solutions by favoring moves along manifolds of constant likelihood, reducing computational overhead.
• The model’s ability to use expert-curated ontologies as prior knowledge enabled topic summaries that were both interpretable and scientifically meaningful to domain experts.