[论文解读] A provable SVD-based algorithm for learning topics in dominant admixture corpus

Trapit Bansal, Chiranjib Bhattacharyya+1 more|arXiv (Cornell University)|Oct 26, 2014

Topic Modeling参考文献 15被引用 36

一句话总结

该论文提出TSVD，一种基于SVD的可证明准确的算法，用于在主导混合语料库中学习主题模型，通过引入主题特异性关键词——即频繁共现且在某一主题中频率严格高于其他主题的词语。在关键词和主导混合的合理假设下，TSVD实现了与词汇表大小无关的有界$l_1$误差，优于真实和半真实语料库中的先前最先进方法。

ABSTRACT

Topic models, such as Latent Dirichlet Allocation (LDA), posit that documents are drawn from admixtures of distributions over words, known as topics. The inference problem of recovering topics from admixtures, is NP-hard. Assuming separability, a strong assumption, [4] gave the first provable algorithm for inference. For LDA model, [6] gave a provable algorithm using tensor-methods. But [4,6] do not learn topic vectors with bounded $l_1$ error (a natural measure for probability vectors). Our aim is to develop a model which makes intuitive and empirically supported assumptions and to design an algorithm with natural, simple components such as SVD, which provably solves the inference problem for the model with bounded $l_1$ error. A topic in LDA and other models is essentially characterized by a group of co-occurring words. Motivated by this, we introduce topic specific Catchwords, group of words which occur with strictly greater frequency in a topic than any other topic individually and are required to have high frequency together rather than individually. A major contribution of the paper is to show that under this more realistic assumption, which is empirically verified on real corpora, a singular value decomposition (SVD) based algorithm with a crucial pre-processing step of thresholding, can provably recover the topics from a collection of documents drawn from Dominant admixtures. Dominant admixtures are convex combination of distributions in which one distribution has a significantly higher contribution than others. Apart from the simplicity of the algorithm, the sample complexity has near optimal dependence on $w_0$, the lowest probability that a topic is dominant, and is better than [4]. Empirical evidence shows that on several real world corpora, both Catchwords and Dominant admixture assumptions hold and the proposed algorithm substantially outperforms the state of the art [5].

研究动机与目标

• 开发一种主题推断算法，在合理假设下可证明地以有界$l_1$误差恢复主题分布。
• 对真实世界文本语料库进行建模，其中文档虽包含多个主题，但由单一主题主导。
• 用更自然且有实证支持的假设（主题特异性关键词）替代强可分性假设（锚定词）。
• 设计一种简单、基于SVD的算法，包含阈值预处理步骤，以确保可证明收敛。
• 实现样本复杂度，对最小主导主题权重$w_0$具有近似最优依赖关系。

提出的方法

• 引入关键词的概念：一组频繁共现且在某一主题中个体频率严格高于其他主题的词语。
• 假设语料库由主导混合生成，即每篇文档中某一主题的权重显著高于其他主题。
• 对文档-词语共现矩阵应用阈值预处理步骤，以分离出高频、主题特异的词语组。
• 对预处理后的矩阵执行截断SVD，提取对应于主题的低秩近似。
• 利用SVD分量恢复主题向量，实现与词汇表大小$d$无关的可证明$l_1$误差界。
• 证明在关键词和主导混合假设下，该算法恢复的主题矩阵误差不会随$d$增长。

实验结果

研究问题

• RQ1 简单的基于SVD的算法是否能在合理假设下实现主题恢复的可证明$l_1$误差界？
• RQ2 主题特异性关键词（在某一主题中频率更高且共现性强）的假设是否能带来优于锚定词的主题恢复效果？
• RQ3 主导混合假设（每篇文档中一个主题占主导）是否能支持可证明且准确的主题推断？
• RQ4 所提算法的样本复杂度如何随最小主导主题权重$w_0$变化？
• RQ5 该算法在真实和半真实语料库上的$l_1$恢复误差是否优于现有最先进方法（如[5]）？

主要发现

• TSVD算法在主题恢复中实现了有界$l_1$误差，且该误差不随词汇表大小$d$增长，而先前工作中的误差随$d$线性增长。
• 在基于真实世界数据集构建的半真实语料库上，TSVD在90%的主题中将$l_1$恢复误差相比最先进方法[5]降低了27%。
• 实证验证表明，关键词和主导混合假设在真实世界语料库中均成立，支持了模型的现实合理性。
• 该算法的样本复杂度对$w_0$（最小主导主题权重）具有近似最优依赖关系，优于[4]。
• 阈值预处理步骤至关重要：它能有效隔离主题特异性词语组，从而实现准确的基于SVD的主题恢复。
• 与锚定词假设相比，使用关键词作为更弱的假设，使模型比基于可分性的方法更具现实性和实证基础。