[论文解读] On some provably correct cases of variational inference for topic models

Pranjal Awasthi, Andrej Risteski|arXiv (Cornell University)|Mar 23, 2015

Machine Learning and Algorithms参考文献 19被引用 8

一句话总结

本文首次在自然假设下对主题模型的变分推断提供了理论分析，证明其可收敛至全局最优解。作者采用新颖的非标准证明技术，将变分更新视为真实参数的带噪声凸组合，证明了在支持基或种子初始化等自然且实用的初始化方法下，收敛具有可证明的保证。

ABSTRACT

Variational inference is a very efficient and popular heuristic used in various forms in the context of latent variable models. It's closely related to Expectation Maximization (EM), and is applied when exact EM is computationally infeasible. Despite being immensely popular, current theoretical understanding of the effectiveness of variaitonal inference based algorithms is very limited. In this work we provide the first analysis of instances where variational inference algorithms converge to the global optimum, in the setting of topic models. More specifically, we show that variational inference provably learns the optimal parameters of a topic model under natural assumptions on the topic-word matrix and the topic priors. The properties that the topic word matrix must satisfy in our setting are related to the topic expansion assumption introduced in (Anandkumar et al., 2013), as well as the anchor words assumption in (Arora et al., 2012c). The assumptions on the topic priors are related to the well known Dirichlet prior, introduced to the area of topic modeling by (Blei et al., 2003). It is well known that initialization plays a crucial role in how well variational based algorithms perform in practice. The initializations that we use are fairly natural. One of them is similar to what is currently used in LDA-c, the most popular implementation of variational inference for topic models. The other one is an overlapping clustering algorithm, inspired by a work by (Arora et al., 2014) on dictionary learning, which is very simple and efficient. While our primary goal is to provide insights into when variational inference might work in practice, the multiplicative, rather than the additive nature of the variational inference updates forces us to use fairly non-standard proof arguments, which we believe will be of general interest.

研究动机与目标

• 为解决尽管变分推断在主题模型中被广泛使用，但缺乏理论保证的问题。
• 确定变分推断收敛至真实模型参数的条件。
• 对考虑变分更新乘法结构的收敛动力学进行形式化分析。
• 利用受现有实现和聚类启发的现实初始化策略，建立可证明的收敛性。
• 弥合变分推断在潜变量模型中实际成功与理论理解之间的鸿沟。

提出的方法

• 将变分推断建模为迭代更新，其结合真实参数与有界误差项，且真实权重随时间占主导地位。
• 将每一步更新视为真实参数与估计误差的带噪声凸组合，利用模型假设确保误差衰减。
• 为每个主题引入“局部”锚定词与锚定文档概念，基于稀疏性与主导性假设推导得出。
• 通过阈值化更新的交替最小化KL散度，以稳定收敛并避免陷入局部最优。
• 应用Chernoff不等式与集中不等式，估算使经验边缘分布近似真实边缘分布所需的文档数量。
• 设计两种初始化方案：一种基于支持识别（类似于LDA-c），另一种基于重叠聚类（受Arora等人2014年工作启发）用于种子初始化。

实验结果

研究问题

• RQ1 在何种条件下，主题模型的变分推断可可证明收敛至全局最优？
• RQ2 如何利用非标准证明技术分析变分推断的乘法更新结构？
• RQ3 初始化在确保收敛至真实模型参数中起何种作用？
• RQ4 主题-词矩阵中的稀疏性与主导性如何促成主题结构的可证明恢复？
• RQ5 为使经验边缘分布以高概率近似真实边缘分布，所需的样本复杂度（文档数量）是多少？

主要发现

• 在主题-词矩阵与主题先验的自然假设下，变分推断可可证明收敛至真实主题模型参数。
• 当使用基于支持的初始化或种子初始化时，收敛性得到保证，这两种方法在实际实现中均自然且广泛使用。
• 分析表明，由于模型假设导致真实值权重占主导，变分更新中的误差项随时间衰减。
• 以高概率，当文档数量满足 $ D = \Omega\left(\frac{K\log^2 K}{\epsilon^2}\right) $ 时，主导文档中的经验主题比例收敛至真实比例。
• 对于判别性词汇，即使存在常见词汇，该方法也能以高概率正确识别其主题支持与主导主题。
• 理论边界表明，存在 $ D_{i,jS} \geq D_i(1-\epsilon)(1-o(1)) $ 个文档，其中某主题占主导且判别性词汇仅出现在该主题中，从而确保可靠识别。