[Paper Review] Progressive EM for Latent Tree Models and Hierarchical Topic Detection

Peixian Chen, Nevin L. Zhang+2 more|arXiv (Cornell University)|Aug 5, 2015

Advanced Text Analysis Techniques3 citations

TL;DR

This paper proposes Progressive EM (PEM) to accelerate Hierarchical Latent Tree Analysis (HLTA), a method for hierarchical topic detection that outperforms LDA-based models in topic coherence and hierarchy quality. By replacing standard EM with a stepwise, localized parameter estimation strategy inspired by the moments method, PEM-HLTA achieves 40x speedup over HLTA—processing the NIPS dataset in 4 minutes versus 17 hours—while matching or exceeding model fit and topic quality, and outperforming state-of-the-art LDA-based methods like nHDP in both efficiency and topic coherence on large-scale data.

ABSTRACT

Hierarchical latent tree analysis (HLTA) is recently proposed as a new method for topic detection. It differs fundamentally from the LDA-based methods in terms of topic definition, topic-document relationship, and learning method. It has been shown to discover significantly more coherent topics and better topic hierarchies. However, HLTA relies on the Expectation-Maximization (EM) algorithm for parameter estimation and hence is not efficient enough to deal with large datasets. In this paper, we propose a method to drastically speed up HLTA using a technique inspired by recent advances in the moments method. Empirical experiments show that our method greatly improves the efficiency of HLTA. It is as efficient as the state-of-the-art LDA-based method for hierarchical topic detection and finds substantially better topics and topic hierarchies.

Motivation & Objective

• To address the computational inefficiency of HLTA, which relies on slow Expectation-Maximization (EM) for parameter estimation and struggles with large-scale datasets.
• To improve the scalability of HLTA without sacrificing topic quality or hierarchical structure fidelity.
• To develop a faster alternative to EM that maintains model fit while drastically reducing training time for latent tree models in topic modeling.
• To enable practical deployment of HLTA on large document collections, such as the New York Times dataset with 300,000 articles.
• To demonstrate that PEM-HLTA achieves comparable or better performance than state-of-the-art LDA-based methods like nHDP, both in speed and topic coherence.

Proposed method

• PEM replaces standard EM by estimating model parameters in a progressive, localized manner, focusing on small subsets of parameters related to only three or four observed variables at a time.
• The method is inspired by the moments method, where parameters are solved via equations involving low-order moments of observed variables, reducing computational complexity.
• PEM-HLTA operates in two phases: hierarchical model construction and progressive parameter estimation using EM-like updates on small, localized parameter groups.
• The algorithm uses a tree-structured Bayesian network representation of the latent tree model (LTM), with binary latent variables forming hierarchical topic clusters.
• Stochastic EM is applied to scale PEM-HLTA further on large datasets by processing data in mini-batches, updating parameters incrementally over multiple passes.
• The method maintains model fit by ensuring per-document log-likelihood values remain comparable to those of standard EM, as confirmed in empirical evaluations.

Experimental results

Research questions

• RQ1 Can a progressive EM approach significantly reduce training time for HLTA without degrading topic coherence or model fit?
• RQ2 How does PEM-HLTA compare to state-of-the-art LDA-based methods like nHDP in terms of topic quality and hierarchical structure on large-scale text data?
• RQ3 To what extent does the progressive parameter estimation strategy preserve model likelihood while improving computational efficiency?
• RQ4 Can PEM-HLTA scale to large datasets such as the New York Times corpus with 300,000 articles and 10,000 words?
• RQ5 Does the use of stochastic EM in the second phase of PEM-HLTA maintain performance while further reducing runtime on massive datasets?

Key findings

• PEM-HLTA reduced training time for the NIPS dataset (1,000 words, <2,000 documents) from 17 hours (HLTA) to just 4 minutes, achieving a 255x speedup.
• On the larger News-5k dataset (10,000 words), PEM-HLTA completed in 365 minutes, while HLTA did not terminate within 3 days, demonstrating superior scalability.
• PEM-HLTA achieved higher average topic coherence scores than all baselines, including nHDP and hLDA, with a score of -12.86 on the New York Times dataset compared to nHDP’s -13.35.
• Per-document log-likelihood values for PEM-HLTA were nearly identical to those of HLTA (e.g., -390 vs. -391 on Nips-1k), confirming that model fit is preserved.
• On the New York Times dataset, PEM-HLTA ran in 11 hours (670 minutes), slightly slower than nHDP (10.5 hours), but produced significantly more coherent topics and better topic hierarchies.
• The method outperformed CorEx, hLDA, and nHDP in topic coherence and model fit, while being more efficient than HLTA and CorEx on large-scale data.