[论文解读] Two SVDs Suffice: Spectral decompositions for probabilistic topic modeling and latent Dirichlet allocation

Animashree Anandkumar, Dean P. Foster+3 more|arXiv (Cornell University)|Apr 30, 2012

Bayesian Methods and Mixture Models被引用 17

一句话总结

该论文提出了一种谱方法——超额相关分析（Excess Correlation Analysis, ECA），通过利用三阶和四阶矩的两次奇异值分解（SVD），高效地恢复主题模型和LDA参数。该方法仅使用三元组统计量即可保证恢复主题向量和主题先验，将计算规模从完整的词汇空间缩减至k×k矩阵。

ABSTRACT

The problem of topic modeling can be seen as a generalization of the clustering problem, in that it posits that observations are generated due to multiple latent factors (e.g., the words in each document are generated as a mixture of several active topics, as opposed to just one). This increased representational power comes at the cost of a more challenging unsupervised learning problem of estimating the topic probability vectors (the distributions over words for each topic), when only the words are observed and the corresponding topics are hidden. We provide a simple and efficient learning procedure that is guaranteed to recover the parameters for a wide class of mixture models, including the popular latent Dirichlet allocation (LDA) model. For LDA, the procedure correctly recovers both the topic probability vectors and the prior over the topics, using only trigram statistics (i.e., third order moments, which may be estimated with documents containing just three words). The method, termed Excess Correlation Analysis (ECA), is based on a spectral decomposition of low order moments (third and fourth order) via two singular value decompositions (SVDs). Moreover, the algorithm is scalable since the SVD operations are carried out on $k imes k$ matrices, where $k$ is the number of latent factors (e.g. the number of topics), rather than in the $d$-dimensional observed space (typically $d \gg k$).

研究动机与目标

• 为解决主题建模中的无监督学习挑战，即潜在主题隐藏而仅能观测到词项。
• 开发一种方法，保证在潜在狄利克雷分配（LDA）中恢复主题概率向量和主题先验。
• 通过在k×k矩阵而非完整的d维观测空间上操作，降低计算成本，其中d ≫ k。
• 仅使用三阶矩（三元组统计量）即可实现参数恢复，即使在短文档上也适用。

提出的方法

• 该方法对词项共现的三阶和四阶矩执行谱分解，以提取潜在主题结构。
• 对这些矩导出的超额相关张量执行两次连续的奇异值分解（SVD）。
• 仅使用三元组统计量即可估计主题概率向量和主题先验分布，这些统计量可从仅含三个词的文档中计算得出。
• 该方法在k×k矩阵上运行，其中k为主题数量，因此可扩展至大规模词汇表。
• 其依赖于主题-词分布线性无关，并满足某些非退化条件以保证可识别性。

实验结果

研究问题

• RQ1 能否通过仅使用三阶和四阶矩的谱方法，恢复LDA参数的完整集合——包括主题向量和主题先验？
• RQ2 是否可能在极少量数据（如仅三词文档）下实现主题模型参数的保证恢复？
• RQ3 如何通过避免在词汇空间中进行全维操作，降低主题建模的计算成本？
• RQ4 三阶和四阶矩在识别混合模型中的潜在因子中起什么作用？

主要发现

• 在主题-词分布满足温和条件时，该方法可保证精确恢复主题概率向量和主题先验。
• 仅需三元组统计量（三阶矩）即可恢复所有LDA参数，从而实现从极短文档中学习。
• 该算法计算效率高，SVD操作在k×k矩阵上执行，而非d×d矩阵，其中d为词汇量。
• 该方法适用于LDA以外的广泛类别的混合模型，包括具有多个潜在因子的模型。