[Paper Review] Chain Rules for Hessian and Higher Derivatives Made Easy by Tensor Calculus

Maciej Skórski|arXiv (Cornell University)|Nov 29, 2019

Tensor decomposition and applications5 references4 citations

TL;DR

This paper presents a simplified, tensor-calculus-based approach to computing chain rules for Hessians and higher-order derivatives in multivariable calculus, using intuitive index pairing and tensor dot products to avoid cumbersome matrix algebra. It demonstrates that by treating derivatives as multidimensional arrays and applying coordinate-free tensor operations, complex higher-order derivatives—especially Hessians under reparametrization—can be derived and computed efficiently and compactly, with verified numerical examples in Python using SymPy.

ABSTRACT

Computing multivariate derivatives of matrix-like expressions in the compact, coordinate free fashion is very important for both theory and applied computations (e.g. optimization and machine learning). The critical components of such computations are \emph{chain and product rules} for derivatives. Although they are taught early in simple scenarios, practical applications involve high-dimensional arrays; in this context it is very hard to find easy accessible and compact explanation. This paper discusses how to relatively simply carry such derivations based on the (simplified as adapted in applied computer science) concept of tensors. Numerical examples in modern Python libraries are provided. This discussion simplifies and illustrates an earlier exposition by Manton (2012).

Motivation & Objective

• To provide a clear, compact, and computationally efficient method for deriving chain rules of Hessians and higher derivatives in multivariable functions.
• To overcome the limitations of traditional matrix-based approaches that fail due to incompatible tensor shapes in high-dimensional arrays.
• To bridge the gap between theoretical multivariable calculus and practical implementation in applied fields like machine learning and optimization.
• To offer a unified, accessible framework using tensor dot products with index pairing, avoiding the need for element-wise differentiation.
• To validate the method with numerical examples using modern Python libraries such as SymPy, highlighting implementation pitfalls due to differing derivative conventions.

Proposed method

• The paper models derivatives as multidimensional arrays (tensors), where the derivative of a function is represented as a tensor with dimensions corresponding to input and output indices.
• It introduces tensor dot products with explicit index pairing: matching indices between tensors (e.g., components of the Jacobian and partial derivatives of the objective) are contracted via summation.
• The method generalizes matrix multiplication to higher-order tensors by allowing flexible pairing of indices across multiple dimensions, preserving shape compatibility.
• It leverages the symmetry of mixed partial derivatives (Schwarz’s Theorem) to simplify the order of operations in tensor contractions.
• The chain rule for the Hessian is derived by applying the product rule to the first derivative $ D(f ullet g) = Df(g) \cdot Dg $, then differentiating again to obtain $ D^2(f \circ g) = (D^2f(g) \cdot Dg \cdot Dg) + (Df(g) \cdot D^2g) $.
• Numerical validation is performed using SymPy’s tensorcontraction and tensorproduct functions, with careful handling of derivative dimension ordering (e.g., derivative index first in SymPy vs. standard convention).

Experimental results

Research questions

• RQ1 How can chain rules for Hessians and higher-order derivatives be expressed in a compact, coordinate-free, and computationally efficient manner for high-dimensional arrays?
• RQ2 What tensor operations enable correct and scalable computation of second-order derivatives when standard matrix multiplication fails due to shape mismatches?
• RQ3 How can index pairing and tensor contraction simplify the derivation of complex derivative expressions in multivariable calculus?
• RQ4 What are the practical implications of differing derivative conventions (e.g., SymPy vs. standard) in numerical implementations of chain rules?
• RQ5 Can tensor calculus provide a more intuitive and less error-prone alternative to traditional matrix-based derivations in optimization and machine learning?

Key findings

• The Hessian of a composed function $ f(g(y)) $ can be computed as $ \mathbf{H}(f \circ g) = \mathbf{J}g^T \cdot \mathbf{H}f(g) \cdot \mathbf{J}g + \sum_{k=1}^{n} \frac{\partial f}{\partial y^k} \cdot \mathbf{H}g^k $, where $ \mathbf{J}g $ is the Jacobian and $ \mathbf{H}g^k $ is the Hessian of the $ k $-th component of $ g $.
• The tensor dot product formulation allows correct handling of higher-order derivatives by pairing indices corresponding to the output dimensions of $ g $ and the input dimensions of $ f $, avoiding shape incompatibility issues.
• Numerical evaluation using SymPy confirms the correctness of the derived chain rule, yielding a diagonal Hessian matrix with entries 2 and 200 for the Rosenbrock function under the given reparametrization.
• The method avoids element-wise differentiation, which is computationally prohibitive for high-dimensional problems, by enabling batch computation through tensor contractions.
• The paper highlights that library-specific conventions (e.g., SymPy placing the derivative dimension first) must be carefully accounted for to ensure correct tensor contractions.
• The use of Einstein summation notation and index pairing provides a natural and scalable framework for expressing complex derivative rules in a compact and interpretable form.