[Paper Review] Scalable Probabilistic Entity-Topic Modeling

Neil Houlsby, Massimiliano Ciaramita|arXiv (Cornell University)|Sep 2, 2013

Topic Modeling36 references3 citations

TL;DR

This paper proposes a scalable probabilistic entity-topic model based on Latent Dirichlet Allocation (LDA), where each topic corresponds to a Wikipedia article and can generate either content words or entity mentions. It uses a hybrid Gibbs sampling and stochastic variational inference framework, enhanced by Wikipedia link graph guidance and a MapReduce-based distributed pipeline, achieving state-of-the-art performance on the Aida-CoNLL dataset with 83.04% micro F1 and efficient training on 5 million documents in under an hour using 1,000 machines.

ABSTRACT

We present an LDA approach to entity disambiguation. Each topic is associated with a Wikipedia article and topics generate either content words or entity mentions. Training such models is challenging because of the topic and vocabulary size, both in the millions. We tackle these problems using a novel distributed inference and representation framework based on a parallel Gibbs sampler guided by the Wikipedia link graph, and pipelines of MapReduce allowing fast and memory-frugal processing of large datasets. We report state-of-the-art performance on a public dataset.

Motivation & Objective

• To address the challenge of large-scale entity disambiguation in text using a topic modeling framework with interpretable, grounded topics.
• To scale probabilistic topic models to millions of topics and large corpora, overcoming computational and memory bottlenecks.
• To integrate structural knowledge from the Wikipedia link graph into the inference process to improve disambiguation accuracy.
• To design a distributed, memory-efficient inference pipeline using MapReduce that supports fast, scalable training and inference.
• To achieve state-of-the-art performance in both scalability and disambiguation accuracy on the Aida-CoNLL benchmark.

Proposed method

• Extends LDA so that each topic is linked to a Wikipedia article and can generate either content words or explicit entity mentions.
• Employs a hybrid inference scheme combining local Gibbs sampling with global stochastic variational inference to balance accuracy and efficiency.
• Guides local Gibbs sampling using the Wikipedia link graph to improve topic consistency and disambiguation quality.
• Designs a MapReduce pipeline where each document is processed as a self-contained packet, serializing model, data, and metadata for distributed inference.
• Uses model sparsity and efficient data serialization to reduce memory overhead and enable fast, scalable processing.
• Applies online learning with parameter updates per document, enabling incremental training on massive datasets.

Experimental results

Research questions

• RQ1 Can a topic model be scaled to millions of topics while maintaining high disambiguation accuracy?
• RQ2 How can the Wikipedia link graph be effectively integrated into the inference process to improve entity disambiguation?
• RQ3 Can a hybrid Gibbs and variational inference framework be made both scalable and efficient in a distributed setting?
• RQ4 What is the performance of such a model on large-scale entity disambiguation benchmarks like Aida-CoNLL?
• RQ5 How does the proposed MapReduce pipeline compare in efficiency and scalability to existing LDA systems?

Key findings

• The model achieved a micro F1 score of 83.04% and macro F1 of 82.84% on the Aida-CoNLL development set, representing state-of-the-art performance.
• Training on 5 million documents with 50 Gibbs sweeps per document completed in approximately one hour using 1,000 machines, demonstrating high scalability.
• The system achieved a memory footprint of only a few hundred megabytes, indicating high memory efficiency despite the large topic space.
• The proposed MapReduce pipeline enabled linear scalability with the number of machines, suggesting strong performance gains with additional resources.
• The model outperformed existing LDA systems in terms of topic space size and training throughput, despite operating in a different regime than standard LDA applications.
• Error analysis revealed limitations in handling multiple mentions with different gold annotations in the same document, due to the bag-of-words assumption, suggesting a need for context-aware weighting in future work.