[论文解读] Recovery guarantee of weighted low-rank approximation via alternating minimization

Yuanzhi Li, Yingyu Liang+1 more|arXiv (Cornell University)|Feb 6, 2016

Sparse and Compressive Sensing Techniques参考文献 24被引用 17

一句话总结

该论文首次为使用非二值确定性权重的交替最小化方法提供了加权低秩矩阵逼近的可证明恢复保证。在温和的相干性和谱隙假设下，即使从随机初始化开始，恢复矩阵与真实矩阵之间误差的谱范数也会随迭代次数呈指数下降，通过一个简单的截断步骤来保持关键矩阵属性。

ABSTRACT

Many applications require recovering a ground truth low-rank matrix from noisy observations of the entries, which in practice is typically formulated as a weighted low-rank approximation problem and solved by non-convex optimization heuristics such as alternating minimization. In this paper, we provide provable recovery guarantee of weighted low-rank via a simple alternating minimization algorithm. In particular, for a natural class of matrices and weights and without any assumption on the noise, we bound the spectral norm of the difference between the recovered matrix and the ground truth, by the spectral norm of the weighted noise plus an additive error that decreases exponentially with the number of rounds of alternating minimization, from either initialization by SVD or, more importantly, random initialization. These provide the first theoretical results for weighted low-rank via alternating minimization with non-binary deterministic weights, significantly generalizing those for matrix completion, the special case with binary weights, since our assumptions are similar or weaker than those made in existing works. Furthermore, this is achieved by a very simple algorithm that improves the vanilla alternating minimization with a simple clipping step. The key technical challenge is that under non-binary deterministic weights, naïve alternating steps will destroy the incoherence and spectral properties of the intermediate solutions, which are needed for making progress towards the ground truth. We show that the properties only need to hold in an average sense and can be achieved by the clipping step. We further provide an alternating algorithm that uses a whitening step that keeps the properties via SDP and Rademacher rounding and thus requires weaker assumptions. This technique can potentially be applied in some other applications and is of independent interest.

研究动机与目标

• 为实际中广泛使用的启发式方法——交替最小化在加权低秩逼近中的理论恢复保证提供支持。
• 将现有的矩阵补全结果推广至具有非二值、确定性权重的一般加权低秩逼近。
• 在最小假设下分析收敛性，包括对噪声无限制且对随机初始化具有鲁棒性。
• 提出一种基于截断的算法，以在不强制每步严格不变的前提下保持相干性和谱属性，这些属性对收敛至关重要。
• 为诸如词嵌入和生物数据分析等应用提供理论基础，这些应用中各条目噪声方差不同。

提出的方法

• 提出一种简单交替最小化算法，通过引入截断步骤以稳定中间解并维持相干性。
• 使用谱范数分析来界定恢复矩阵与真实矩阵之间的误差。
• 提出一种新颖的分析框架，表明与真实矩阵的距离每轮迭代呈指数下降。
• 通过截断实现平均意义上的相干性和谱属性保持，而非要求每步严格不变。
• 提供一种替代算法，利用半定规划进行白化处理并结合Rademacher取样，以在更弱假设下保持属性。
• 分析权重矩阵的谱隙对收敛性的影响，并在词嵌入语料库上对假设进行经验验证。

实验结果

研究问题

• RQ1 对于一般非二值权重的加权低秩逼近，交替最小化能否在理论上保证恢复真实矩阵？
• RQ2 该算法是否能从随机初始化开始收敛，而无需依赖SVD初始化？
• RQ3 误差随迭代次数如何衰减，能否被加权噪声的谱范数加上一个指数衰减项所界定？
• RQ4 权重矩阵的谱隙在确保收敛性和稳定性方面起什么作用？
• RQ5 该算法能否在不强制每步严格不变的前提下，保持中间解的相干性和谱属性？

主要发现

• 恢复矩阵与真实矩阵之间误差的谱范数，被界定为加权噪声的谱范数加上一个随交替最小化迭代次数呈指数衰减的附加误差项。
• 在温和假设下，恢复保证既适用于SVD初始化，也适用于更关键的随机初始化。
• 截断步骤确保相干性和谱属性在平均意义上得以保持，从而在非凸条件下实现收敛。
• 该理论框架推广了先前的矩阵补全结果，并适用于噪声方差非均匀的场景，如词嵌入和生物数据。
• 在大规模语料库上的实证验证表明，截断后的共现权重（例如，f(X_ij) = max(X_ij, 100)）能产生显著的谱隙，支持理论假设。
• 基于SDP白化和Rademacher取样的替代算法所需假设更弱，且对其他应用亦具独立研究价值。