[Paper Review] Linnea: Automatic Generation of Efficient Linear Algebra Programs

Henrik Barthels, Christos Psarras+1 more|arXiv (Cornell University)|Dec 30, 2019

Parallel Computing and Optimization Techniques50 references4 citations

TL;DR

Linnea is a code generator that automatically transforms high-level linear algebra expressions into optimized sequences of BLAS and LAPACK kernel calls. Using symbolic rewrite rules and a best-first search algorithm, it generates efficient, correct-by-construction code that outperforms high-level languages like MATLAB, Julia, Eigen, and Armadillo by up to 10× in 125 test problems.

ABSTRACT

The translation of linear algebra computations into efficient sequences of library calls is a non-trivial task that requires expertise in both linear algebra and high-performance computing. Almost all high-level languages and libraries for matrix computations (e.g., Matlab, Eigen) internally use optimized kernels such as those provided by BLAS and LAPACK; however, their translation algorithms are often too simplistic and thus lead to a suboptimal use of said kernels, resulting in significant performance losses. In order to combine the productivity offered by high-level languages, and the performance of low-level kernels, we are developing Linnea, a code generator for linear algebra problems. As input, Linnea takes a high-level description of a linear algebra problem; as output, it returns an efficient sequence of calls to high-performance kernels. Linnea uses a custom best-first search algorithm to find a first solution in less than a second, and increasingly better solutions when given more time. In 125 test problems, the code generated by Linnea almost always outperforms Matlab, Julia, Eigen and Armadillo, with speedups up to and exceeding 10x.

Motivation & Objective

• To address the performance gap between high-level linear algebra abstractions and low-level optimized kernels like BLAS and LAPACK.
• To automate the translation of mathematical expressions into efficient, correct sequences of library calls without requiring expert knowledge in high-performance computing.
• To reduce the manual effort and error-prone nature of hand-optimizing linear algebra code for performance.
• To enable both non-experts and experts to generate performant code by leveraging symbolic optimization and domain-specific knowledge.
• To improve productivity in scientific computing by combining high-level expressiveness with low-level performance.

Proposed method

• Accepts high-level linear algebra expressions as input, including matrices, vectors, scalars, and operations like addition, multiplication, transposition, and inversion.
• Uses symbolic manipulation and rewrite rules to apply algebraic identities (e.g., distributivity, subexpression reuse) for performance optimization.
• Employs a custom best-first search algorithm to explore the space of possible code sequences, prioritizing efficient solutions.
• Generates Julia code as output, leveraging Julia’s efficient BLAS/LAPACK wrappers and extensibility for high performance.
• Supports matrix properties (e.g., SPD, triangular, symmetric) as annotations to guide kernel selection and optimization.
• Uses a cost model to guide search, with merging of search branches to improve solution quality over time.

Experimental results

Research questions

• RQ1 Can a symbolic code generator automatically produce more efficient linear algebra code than high-level languages like MATLAB, Julia, Eigen, and Armadillo?
• RQ2 How effective is a best-first search strategy with symbolic rewrite rules in discovering performant code sequences?
• RQ3 To what extent can automatic optimization of linear algebra expressions reduce performance gaps caused by naive translation?
• RQ4 How does the inclusion of matrix properties (e.g., SPD, triangular) improve code generation quality?
• RQ5 What is the trade-off between generation time and code quality in the context of automatic linear algebra code synthesis?

Key findings

• In 125 test problems, Linnea-generated code consistently outperformed MATLAB, Julia, Eigen, and Armadillo, with speedups up to and exceeding 10×.
• Linnea found a solution in under one second and continued improving it with more time, demonstrating efficient search in the optimization space.
• The use of symbolic rewrite rules enabled correct-by-construction code generation, avoiding correctness issues common in manual optimization.
• The integration of matrix properties (e.g., SPD, triangular) allowed Linnea to select optimal kernels such as Cholesky or specialized solvers, reducing computational cost.
• The generated code was more efficient than both naive and recommended implementations in libraries like Eigen and Armadillo, even when those used optimized functions.
• The cost model effectively guided the search, and merging search branches improved solution quality over time.