[论文解读] The Linear Algebra Mapping Problem

Christos Psarras, Henrik Barthels+1 more|arXiv (Cornell University)|Nov 21, 2019

Parallel Computing and Optimization Techniques参考文献 38被引用 7

一句话总结

本文提出了线性代数映射问题，识别了高级线性代数代码与低级优化库之间的性能差距。它评估了 MATLAB、Julia 和 NumPy 等语言将高级矩阵表达式转换为高效内核调用的效率，揭示了当前编译器和解释器存在显著的次优性，并提出了改进语言和库设计以缩小这一差距的指导原则。

ABSTRACT

We observe a disconnect between the developers and the end users of linear algebra libraries. On the one hand, the numerical linear algebra and the high-performance communities invest significant effort in the development and optimization of highly sophisticated numerical kernels and libraries, aiming at the maximum exploitation of both the properties of the input matrices, and the architectural features of the target computing platform. On the other hand, end users are progressively less likely to go through the error-prone and time consuming process of directly using said libraries by writing their code in C or Fortran; instead, languages and libraries such as Matlab, Julia, Eigen and Armadillo, which offer a higher level of abstraction, are becoming more and more popular. Users are given the opportunity to code matrix computations with a syntax that closely resembles the mathematical description; it is then a compiler or an interpreter that internally maps the input program to lower level kernels, as provided by libraries such as BLAS and LAPACK. Unfortunately, our experience suggests that in terms of performance, this translation is typically vastly suboptimal. In this paper, we first introduce the Linear Algebra Mapping Problem, and then investigate how effectively a benchmark of test problems is solved by popular high-level programming languages. Specifically, we consider Matlab, Octave, Julia, R, Armadillo (C++), Eigen (C++), and NumPy (Python); the benchmark is meant to test both standard compiler optimizations such as common subexpression elimination and loop-invariant code motion, as well as linear algebra specific optimizations such as optimal parenthesization of a matrix product and kernel selection for matrices with properties. The aim of this study is to give concrete guidelines for the development of languages and libraries that support linear algebra computations.

研究动机与目标

• 识别高级线性代数抽象与低级优化库（如 BLAS 和 LAPACK）之间的性能差距。
• 研究主流高级语言在多大程度上将高级矩阵表达式映射到高效的低级内核。
• 评估通用编译器优化与线性代数特定优化对性能的影响。
• 为未来线性代数计算语言和库的改进提供具体指导原则。

提出的方法

• 本研究将线性代数映射问题定义为将高级矩阵表达式高效转换为最优低级内核调用的挑战。
• 设计了一套测试问题基准，用于评估对常见子表达式消除、循环不变代码移动以及矩阵特定优化（如最优括号化）的支持情况。
• 在多种语言中执行该基准测试：MATLAB、Octave、Julia、R、Armadillo（C++）、Eigen（C++）和 NumPy（Python）。
• 通过将执行时间与已知最优内核使用情况进行比较，衡量性能，评估通用优化与线性代数特定优化的效果。
• 分析重点在于编译器和解释器在多大程度上能将数学符号映射到高效的数值内核，特别是针对具有特殊性质的矩阵。

实验结果

研究问题

• RQ1 高级语言在多大程度上将矩阵表达式映射到最优的 BLAS 和 LAPACK 内核？
• RQ2 当前编译器和解释器在多大程度上利用了线性代数代码中的常见子表达式消除和循环不变代码移动？
• RQ3 这些语言在为具有特定性质的矩阵选择合适内核方面有多高效？
• RQ4 从高级语法到低级内核的转换过程中，性能瓶颈是什么？

主要发现

• 从高级矩阵表达式到低级内核的转换通常严重次优，导致显著的性能下降。
• 即使使用复杂的高级语言，最优执行与实际执行之间的性能差距依然显著。
• 常见子表达式消除和循环不变代码移动在大多数线性代数上下文中并未被有效应用。
• 矩阵乘积的最优括号化以及基于矩阵特性的内核选择，通常被当前编译器和解释器所忽略。