[论文解读] Recovering Structured Probability Matrices

Qingqing Huang, Sham M. Kakade+2 more|arXiv (Cornell University)|Feb 21, 2016

Machine Learning and Algorithms参考文献 48被引用 4

一句话总结

该论文提出了一种高效算法，通过截断SVD和加权归一化进行结构化估计，从稀疏观测中恢复低秩概率矩阵。研究证明，当矩阵的秩为常数时，Θ(M) 个样本在信息论上是最优的，且证明了测试基本属性（如秩或均匀性）的子线性样本复杂度是不可能的。

ABSTRACT

We consider the problem of accurately recovering a matrix B of size M by M , which represents a probability distribution over M2 outcomes, given access to an observed matrix of "counts" generated by taking independent samples from the distribution B. How can structural properties of the underlying matrix B be leveraged to yield computationally efficient and information theoretically optimal reconstruction algorithms? When can accurate reconstruction be accomplished in the sparse data regime? This basic problem lies at the core of a number of questions that are currently being considered by different communities, including building recommendation systems and collaborative filtering in the sparse data regime, community detection in sparse random graphs, learning structured models such as topic models or hidden Markov models, and the efforts from the natural language processing community to compute "word embeddings". Our results apply to the setting where B has a low rank structure. For this setting, we propose an efficient algorithm that accurately recovers the underlying M by M matrix using Theta(M) samples. This result easily translates to Theta(M) sample algorithms for learning topic models and learning hidden Markov Models. These linear sample complexities are optimal, up to constant factors, in an extremely strong sense: even testing basic properties of the underlying matrix (such as whether it has rank 1 or 2) requires Omega(M) samples. We provide an even stronger lower bound where distinguishing whether a sequence of observations were drawn from the uniform distribution over M observations versus being generated by an HMM with two hidden states requires Omega(M) observations. This precludes sublinear-sample hypothesis tests for basic properties, such as identity or uniformity, as well as sublinear sample estimators for quantities such as the entropy rate of HMMs.

研究动机与目标

• 解决从其条目独立同分布的稀疏样本中准确重构M×M低秩概率矩阵的挑战。
• 理解结构假设（特别是低秩）如何降低矩阵恢复和属性测试的样本复杂度。
• 弥合稀疏估计中结构化概率矩阵的计算可行性与信息论极限之间的差距。
• 建立紧致的下界，表明测试基本属性（如秩或均匀性）的子线性样本复杂度是不可能的。
• 为协同过滤、词嵌入和HMM学习等应用提供一种实用且理论最优的算法。

提出的方法

• 该方法通过一个对角矩阵D进行加权归一化，以在稀疏区域稳定估计，将观测到的计数矩阵转化为一个条件良好的形式。
• 对归一化后的矩阵应用截断奇异值分解（SVD），以恢复主导的秩-1分量，从而利用低秩结构。
• 通过将投影到子空间V上来隔离信号与噪声，随后对投影矩阵进行秩-1逼近。
• 基于经验边缘概率对条目进行分桶，以控制不同概率质量尺度下的估计误差。
• 结合大偏差不等式和谱范数界，推导出恢复向量的高概率误差保证。
• 最终估计通过反转归一化并选择使ℓ₁误差最小的奇异向量符号获得。

实验结果

研究问题

• RQ1 我们能否以计算高效且信息论最优的方式，从O(M)个样本中恢复低秩概率矩阵？
• RQ2 测试结构化概率矩阵的基本属性（如秩1与秩2）的根本样本复杂度极限是什么？
• RQ3 对于HMM或主题模型等结构化矩阵，估计或测试其属性的子线性样本复杂度是否可能？
• RQ4 底层矩阵的结构（如低秩、良好条件）如何影响估计和属性测试的样本复杂度？
• RQ5 我们能否设计一种实用算法，在稀疏数据环境中达到信息论极限？

主要发现

• 所提出的算法使用N = O(M / (w_min^4 * ε^9)) 个样本，以ℓ₁误差有界于ϵ的方式恢复底层低秩概率矩阵，实现了在常数因子内的最优样本复杂度。
• Θ(M)的样本复杂度在信息论上是最优的，因为即使测试矩阵是否具有秩1或秩2，也至少需要Ω(M)个样本。
• 更强的下界表明，区分均匀分布与良好条件的两状态HMM需要Ω(M)个样本，从而排除了子线性样本量的假设检验。
• 该算法实现了高概率恢复，误差衰减为O(√(M/N) + e^(-e^{k₀-2} - k₀)/2 + (1/(e^{k₀} w_min²))^{1/4})，在足够样本下可任意减小。
• 该方法在实践中可行，适用于词嵌入学习和稀疏数据环境下的协同过滤等实际问题。
• 分析建立了恢复向量ℓ₁误差的紧致界，表明在适当的采样条件下，其会收敛到真实信号。