[论文解读] Superfast Accurate Low Rank Approximation

Victor Y. Pan, Qi Luan+2 more|arXiv (Cornell University)|Nov 3, 2016

Sparse and Compressive Sensing Techniques参考文献 66被引用 6

一句话总结

本文提出了两种新颖的低秩矩阵逼近算法，可在平均输入下实现超快性能——使用的浮点运算次数少于矩阵元素的个数。通过简化并增强现有的交叉逼近技术，引入新型稀疏乘数，该方法在平均输入下展现出高经验效率与理论保证，同时通过预处理识别并缓解了失败情况。

ABSTRACT

Low Rank Approximation is among most fundamental subjects of numerical linear algebra having important applications to various areas of modern computing and %they range from machine learning theory and %neural networks to data mining and analysis. The known algorithms compute such approximations by using more flops than the input matrix has entries, but we prove that much fewer flops than entries are sufficient in the case of the average input (flop stands for floating point arithmetic operation). We prove this twice -- for the solutions by means of two distinct algorithms, and we analyze them by applying two different approaches. Our analysis of both algorithms is quite involved, but we devise them mostly by simplifying, combining, and ameliorating the known techniques, although we propose some technical novelties for further enhancing the performance of the popular Cross-Approximation Algorithms. They are highly efficient empirically, and we prove that they are efficient for the average input. We specify some narrow classes of hard inputs for which the presented algorithms fail with high probability even when we randomize them, but we narrow such classes further by means of preprocessing with new sparse and structured multipliers. The average complexity estimates do not cover many realistic input classes, but our formal analysis is in good accordance with the results of our tests applied to benchmark inputs from discretized PDEs and Integral quations and to random inputs. Our work should already be of practical value but also leads to research challenges. At the end we list some of them, propose two novel extensions of our progress -- to the acceleration of the Fast Multipole Method and Conjugate Gradient algorithms, and explore and slightly extend the recent techniques of Osinsky, which enhance the output accuracy of CUR Approximation.

研究动机与目标

• 解决现有低秩逼近算法效率低下的问题，即所需浮点运算次数超过矩阵元素个数。
• 开发在平均输入下相对于输入规模实现亚线性浮点运算复杂度的算法。
• 通过新颖的技术改进，提升交叉逼近算法的实际性能与理论保证。
• 通过结构化稀疏预处理识别并缓解标准算法易失效的特定困难输入类别。
• 将该方法的适用范围扩展至离散化PDE和积分方程等实际基准问题。

提出的方法

• 基于交叉逼近技术设计两种不同的算法，针对平均输入行为进行优化。
• 采用两种不同的分析方法，严格证明算法在平均输入下可实现亚线性浮点运算复杂度。
• 引入新型稀疏且结构化的乘数对输入进行预处理，以降低在困难输入类别上失败的概率。
• 简化并整合已知的交叉逼近技术，结合新颖的改进方法以提升性能与稳定性。
• 利用近期CUR逼近技术的进展，特别是Osinsky的工作，以提高输出精度。
• 在PDE、积分方程和随机矩阵等基准输入上进行实验验证，以支持理论结论。

实验结果

研究问题

• RQ1 能否在平均情况下，使用少于矩阵元素个数的浮点运算次数完成低秩逼近？
• RQ2 如何修改交叉逼近算法，使其在保持高精度的同时实现亚线性复杂度？
• RQ3 哪些输入类别会导致标准交叉逼近算法失效，又该如何缓解？
• RQ4 理论复杂度界在多大程度上与真实输入类别上的实际性能一致？
• RQ5 所提方法在多大程度上可扩展以加速其他数值算法（如快速多体法和共轭梯度法）？

主要发现

• 所提算法在平均输入下实现了相对于矩阵规模的亚线性浮点运算复杂度，证明了少于矩阵元素个数的运算次数已足够。
• 理论分析确认两种算法在平均情况下均高效，且通过两种不同的数学方法验证了结果。
• 在离散化PDE和积分方程上的实证测试表明，理论预测与实际性能高度一致。
• 存在一个狭窄的困难输入类别，算法在该类上以高概率失败，但通过使用稀疏结构化乘数进行预处理，该问题显著缓解。
• 该方法在实证中表现出极高的效率，已具备实际应用价值，且有潜力扩展以加速快速多体法和共轭梯度求解器。
• 本工作为通过整合Osinsky的最新技术提升CUR逼近精度奠定了基础。