[论文解读] Randomized Numerical Linear Algebra : A Perspective on the Field With an Eye to Software

Riley Murray, James Demmel+11 more|arXiv (Cornell University)|Feb 22, 2023

Stochastic Gradient Optimization Techniques被引用 16

一句话总结

本文综述 RandNLA，主张标准 RandBLAS 和 RandLAPACK 库，并为随机线性代数算法与软件勾勒出一个务实、易于实现的路线图。

ABSTRACT

Randomized numerical linear algebra - RandNLA, for short - concerns the use of randomization as a resource to develop improved algorithms for large-scale linear algebra computations. The origins of contemporary RandNLA lay in theoretical computer science, where it blossomed from a simple idea: randomization provides an avenue for computing approximate solutions to linear algebra problems more efficiently than deterministic algorithms. This idea proved fruitful in the development of scalable algorithms for machine learning and statistical data analysis applications. However, RandNLA's true potential only came into focus upon integration with the fields of numerical analysis and "classical" numerical linear algebra. Through the efforts of many individuals, randomized algorithms have been developed that provide full control over the accuracy of their solutions and that can be every bit as reliable as algorithms that might be found in libraries such as LAPACK. Recent years have even seen the incorporation of certain RandNLA methods into MATLAB, the NAG Library, NVIDIA's cuSOLVER, and SciKit-Learn. For all its success, we believe that RandNLA has yet to realize its full potential. In particular, we believe the scientific community stands to benefit significantly from suitably defined "RandBLAS" and "RandLAPACK" libraries, to serve as standards conceptually analogous to BLAS and LAPACK. This 200-page monograph represents a step toward defining such standards. In it, we cover topics spanning basic sketching, least squares and optimization, low-rank approximation, full matrix decompositions, leverage score sampling, and sketching data with tensor product structures (among others). Much of the provided pseudo-code has been tested via publicly available MATLAB and Python implementations.

研究动机与目标

• 推动 RandNLA 作为大规模线性代数问题的可扩展方法。
• 阐明随机性如何揭示隐藏结构以实现更快、可控的精度。
• 提出面向软件的框架（RandBLAS/RandLAPACK），以标准化实现与部署。

提出的方法

• 描述草绘（sketching）作为线性代数中降维的核心随机化技术。
• 将草绘算子（密集、稀疏和基于变换）及其性质进行分类。
• 勾勒驱动级算法（最小二乘、优化和低秩近似）及其计算例程。
• 讨论有限精度算术和数据移动如何影响算法性能。
• 倡导具有明确 API 的模块化软件体系结构（用于草绘的 RandBLAS；用于更高层问题的 RandLAPACK）。
• 提供伪代码、附录，以及经过测试的 MATLAB/Python 实现以支持软件采用。

实验结果

研究问题

• RQ1 随机草绘技术如何标准化以形成可靠、高性能的线性代数软件库？
• RQ2 RandNLA 基于的算法必须在常见问题类别（LS、优化、低秩近似）上满足哪些核心设计原则和性能保证？
• RQ3 为最大化跨硬件和软件生态系统的可移植性、效率和易用性，RandBLAS 和 RandLAPACK 应如何组织？
• RQ4 有限精度算术和数据移动对 RandNLA 方法的精度与性能有何实际影响？
• RQ5 需要哪些关键的经验基准和软件抽象来加速 RandNLA 在科学计算与 ML 流水线中的采用？

主要发现

• RandNLA 通过利用随机草绘，在许多被高估计或高维线性代数问题上提供近线性或线性时间的方法。
• 随机化减少数据移动，与经典确定性方法相比可实现墙钟时间加速。
• 用户可以通过可调的随机化算法在精度和计算成本之间权衡，在大规模问题上具有可预测行为。
• 草绘分布和基本随机数生成在固定种子后能实现确定性行为，促进可重复性。
• 作者提倡两库生态系统（RandBLAS 用于草绘，RandLAPACK 用于驱动）以标准化并加速 RandNLA 软件开发。
• 公开领域实现（MATLAB/Python）伴随理论与算法发展，帮助推广采用。