[Paper Review] Algorithmic Differentiation of Linear Algebra Functions with Application in Optimum Experimental Design (Extended Version)

Sebastian F. Walter, Lutz Lehmann|arXiv (Cornell University)|Jan 11, 2010

Control Systems and Identification6 references3 citations

TL;DR

This paper presents novel algorithmic differentiation (AD) algorithms for higher-order derivatives of rectangular QR and symmetric eigenvalue decompositions using univariate Taylor polynomial propagation (UTPM) in both forward and reverse modes. The approach treats linear algebra functions as elementary operations, achieving near-machine precision accuracy and enabling efficient gradient computation in optimum experimental design problems.

ABSTRACT

We derive algorithms for higher order derivative computation of the rectangular $QR$ and eigenvalue decomposition of symmetric matrices with distinct eigenvalues in the forward and reverse mode of algorithmic differentiation (AD) using univariate Taylor propagation of matrices (UTPM). Linear algebra functions are regarded as elementary functions and not as algorithms. The presented algorithms are implemented in the BSD licensed AD tool exttt{ALGOPY}. Numerical tests show that the UTPM algorithms derived in this paper produce results close to machine precision accuracy. The theory developed in this paper is applied to compute the gradient of an objective function motivated from optimum experimental design: $ abla_x Φ(C(J(F(x,y))))$, where $Φ= \{λ_1 : λ_1 C\}$, $C = (J^T J)^{-1}$, $J = \frac{\dd F}{\dd y}$ and $F = F(x,y)$.

Motivation & Objective

• To develop efficient, high-accuracy algorithmic differentiation methods for key linear algebra functions like QR and eigenvalue decompositions.
• To treat linear algebra routines as elementary functions in AD, avoiding differentiation of their internal algorithms.
• To enable accurate gradient computation in optimum experimental design problems involving matrix functions.
• To implement and benchmark the derived algorithms in the open-source ALGOPY AD tool.
• To demonstrate the superiority of UTPM-based AD over traditional AD differentiation of linear algebra routines in terms of performance and memory efficiency.

Proposed method

• Uses univariate Taylor polynomial propagation (UTPM) to represent matrix-valued functions as polynomial expansions in a dual variable t.
• Derives forward and reverse mode AD algorithms for rectangular QR decomposition and symmetric eigenvalue decomposition with distinct eigenvalues.
• Applies pullback and pushforward operations to propagate derivative information through matrix decompositions using lifted function representations.
• Treats matrix decompositions as atomic operations, bypassing the need to differentiate their underlying iterative algorithms.
• Implements the algorithms in ALGOPY, a BSD-licensed AD tool, using a computational graph with doubly linked nodes instead of a sequential tape.
• Employs matrix Taylor expansions to compute higher-order derivatives efficiently, reducing memory usage and improving performance.

Experimental results

Research questions

• RQ1 How can higher-order derivatives of rectangular QR decomposition be computed efficiently using algorithmic differentiation?
• RQ2 What is the structure of the adjoint (pullback) for symmetric eigenvalue decomposition with distinct eigenvalues in reverse mode AD?
• RQ3 How does UTPM-based AD compare to differentiating the underlying linear algebra algorithms in terms of accuracy and performance?
• RQ4 Can treating linear algebra functions as elementary operations in AD yield better numerical stability and efficiency in scientific computing applications?
• RQ5 What is the practical impact of UTPM-based AD on gradient computation in optimum experimental design problems?

Key findings

• The UTPM-based AD algorithms achieve numerical accuracy close to machine precision, with a relative error of approximately 4.4×10⁻¹⁵ when compared to symbolic derivatives.
• The push forward runtime ratio (UTPM AD over normal function evaluation) is approximately 11.79 for QR decomposition and 11.88 for eigenvalue decomposition, indicating moderate overhead.
• The UTPM implementation in ALGOPY outperforms UTPS differentiation via PYADOLC, especially for large matrices, due to better memory access and computational structure.
• The approach reduces memory requirements in reverse mode AD to O(D) by avoiding storage of intermediate values, leveraging analytical derivative formulas.
• The method enables stable and accurate gradient computation for the objective function Φ = largest eigenvalue of (JᵀJ)⁻¹ in optimum experimental design.
• The implementation in ALGOPY is publicly available on GitHub and serves as a testbed for verified, portable AD algorithms.