[Paper Review] On the Higher-Order Derivatives of Spectral Functions: Two Special Cases

Hristo S. Sendov|ArXiv.org|Apr 19, 2004

Matrix Theory and Algorithms11 references3 citations

TL;DR

This paper presents a unified, tensor-based framework for computing higher-order derivatives of spectral functions—functions of eigenvalues of symmetric matrices—using generalized Hadamard products. It derives explicit formulas for the k-th derivative in two key cases: (1) when the underlying function is k-times differentiable and the matrix has distinct eigenvalues, and (2) when the function is separable and symmetric, valid for any symmetric matrix. The key contribution is a streamlined, calculus-like derivation of the Hessian of general spectral functions, significantly simplifying prior results.

ABSTRACT

In this work we use the tensorial language developed in [8] and [9] to differentiate functions of eigenvalues of symmetric matrices. We describe the formulae for the k-th derivative of such functions in two cases. The first case concerns the derivatives of the composition of an arbitrary differentiable function with the eigenvalues at a matrix with distinct eigenvalues. The second development describes the derivatives of the composition of a separable symmetric function with the eigenvalues at an arbitrary symmetric matrix. In the concluding section we re-derive the formula for the Hessian of a general spectral function at an arbitrary point. Our approach leads to a shorter, streamlined derivation than the original in [6]. The language we use, based on the generalized Hadamard product, allows us to view the differentiation of spectral functions as a routine calculus-type procedure.

Motivation & Objective

• To develop a systematic, calculus-style method for computing higher-order derivatives of spectral functions of symmetric matrices.
• To generalize prior results on gradients and Hessians of spectral functions to arbitrary order k.
• To provide a unified framework based on tensor analysis and generalized Hadamard products that treats eigenvalue dependence and orthogonal conjugation as separate, modular components.
• To re-derive the Hessian of a general spectral function in a shorter, more transparent way than previous approaches.
• To establish conditions under which the k-th derivative formula simplifies, particularly for separable symmetric functions.

Proposed method

• Uses a tensorial language based on generalized Hadamard products to express higher-order derivatives of spectral functions.
• Expresses the k-th derivative of $ f \circ \lambda $ as $ \nabla^k(f \circ \lambda)(X) = V \left( \sum_{\sigma \in P^k} \mathrm{Diag}^\sigma \mathcal{A}_\sigma(\lambda(X)) \right) V^T $, where $ V $ diagonalizes $ X $.
• Defines the operators $ \mathrm{Diag}^\sigma $ and $ \mathcal{A}_\sigma $ in terms of partial derivatives of $ f $, independent of eigenvalues.
• Applies the framework to two special cases: (1) $ f $ $ k $-times differentiable with distinct eigenvalues, and (2) $ f $ separable symmetric with arbitrary symmetric matrix.
• Uses block-constant tensors and differential limits to derive the Hessian, leveraging properties of $ \nabla f $ and its derivatives.
• Establishes continuity and differentiability of $ f \circ \lambda $ via analysis of $ \mathcal{A}_\sigma $ and their limits under perturbations.

Experimental results

Research questions

• RQ1 How can higher-order derivatives of spectral functions be systematically computed using a unified tensor-based formalism?
• RQ2 What is the structure of the k-th derivative of $ f \circ \lambda $ when $ f $ is $ k $-times differentiable and the matrix has distinct eigenvalues?
• RQ3 How does the formula for the k-th derivative simplify when $ f $ is a separable symmetric function?
• RQ4 Can the Hessian of a general spectral function be re-derived in a more concise and transparent way using this framework?
• RQ5 What conditions ensure the differentiability and continuity of $ f \circ \lambda $ in terms of the underlying symmetric function $ f $?

Key findings

• The k-th derivative of $ f \circ \lambda $ at a symmetric matrix $ X $ with distinct eigenvalues is given by $ V \left( \sum_{\sigma \in P^k} \mathrm{Diag}^\sigma \mathcal{A}_\sigma(\lambda(X)) \right) V^T $, where $ \mathcal{A}_\sigma $ depends only on partial derivatives of $ f $ up to order $ k $.
• For separable symmetric functions, all $ \mathcal{A}_\sigma $ are identical across permutations $ \sigma $, leading to a significant simplification of the k-th derivative formula.
• The Hessian of a general spectral function is re-derived in a shorter, more intuitive way than in [6], using the generalized Hadamard product formalism.
• The framework shows that $ f \circ \lambda $ is $ C^k $ if and only if $ f $ is $ C^k $, and the differentiability is preserved under the tensor-based differentiation process.
• The method treats eigenvalue dependence and orthogonal conjugation as separable components, enabling a routine, calculus-like computation of derivatives.
• The use of block-constant tensors and differential limits allows for a clean, transparent derivation of the Hessian, avoiding the complex algebraic manipulations of earlier approaches.