[Paper Review] Automatic Differentiation for Complex Valued SVD

Zhou‐Quan Wan, Shi‐Xin Zhang|arXiv (Cornell University)|Sep 4, 2019

Computational Physics and Python Applications1 references4 citations

TL;DR

This paper derives the reverse-mode automatic differentiation (AD) formula for complex-valued singular value decomposition (SVD), filling a critical gap in AD infrastructure for complex tensor networks. The key contribution is a novel term in the gradient formula—$ar{oldsymbol{A}}_d = rac{1}{2} U S^{-1} (oldsymbol{L}^ op - oldsymbol{L}) V^ op$—that arises due to complex analytic structure and has no counterpart in real SVD, enabling accurate backpropagation in quantum physics and machine learning applications involving complex tensors.

ABSTRACT

In this note, we report the back propagation formula for complex valued singular value decompositions (SVD). This formula is an important ingredient for a complete automatic differentiation(AD) infrastructure in terms of complex numbers, and it is also the key to understand and utilize AD in tensor networks.

Motivation & Objective

• Address the lack of reverse-mode automatic differentiation (AD) support for complex-valued singular value decomposition (SVD), a critical gap in AD infrastructure for complex tensor networks.
• Enable gradient computation in optimization pipelines involving complex matrices, especially in quantum physics and machine learning where complex-valued tensors are common.
• Provide a mathematically rigorous derivation of the backpropagation formula for complex SVD, including the treatment of gauge invariance and non-holomorphic dependencies.
• Ensure the derived formula is numerically stable and compatible with existing AD frameworks by identifying the correct gradient expressions for complex matrix functions.
• Facilitate the integration of SVD into differentiable tensor network models by supplying a complete and correct gradient formula for complex-valued operations.

Proposed method

• Derives the reverse-mode AD gradient for complex SVD using complex calculus, focusing on real-valued loss functions and defining gradients via Wirtinger derivatives.
• Applies the chain rule to the SVD decomposition $A = USV^ op$, expressing differentials of $U$, $S$, and $V$ in terms of skew-Hermitian matrices due to unitarity constraints.
• Introduces auxiliary matrices $oldsymbol{J} = F igcirc (U^ op ar{U})$, $oldsymbol{K} = F igcirc (V^ op ar{V})$, and $oldsymbol{L} = I igcirc (V^ op ar{V})$, where $F_{ij} = rac{1}{s_j^2 - s_i^2}$ for $i eq j$, to capture off-diagonal contributions.
• Identifies a new term $ar{oldsymbol{A}}_d = rac{1}{2} U S^{-1} (oldsymbol{L}^ op - oldsymbol{L}) V^ op$ arising from the non-holomorphic nature of real-valued functions of complex variables.
• Uses gauge invariance and trace identities to derive consistency conditions, ensuring the gradient is independent of phase ambiguities in the SVD decomposition.
• Transforms the derivative expressions into gradient form via $ar{oldsymbol{oldsymbol{A}}} = 2 (ar{oldsymbol{A}}^*)$ for real-valued loss functions, yielding the final formula in Equation (1).

Experimental results

Research questions

• RQ1 What is the correct reverse-mode automatic differentiation formula for complex-valued SVD, particularly in the context of real-valued loss functions?
• RQ2 How does the gradient of a complex SVD differ from its real-valued counterpart due to the non-holomorphic nature of real-valued functions of complex variables?
• RQ3 What role does gauge invariance play in the derivation of gradients for complex SVD, and how does it constrain the form of the gradient expressions?
• RQ4 Why is the term $rac{1}{2} U S^{-1} (oldsymbol{L}^ op - oldsymbol{L}) V^ op$ necessary and absent in real SVD, and what physical or mathematical principle underlies its emergence?
• RQ5 How can one construct a valid test loss function that ensures the new gradient term is activated and correctly computed in practice?

Key findings

• The paper derives a complete reverse-mode automatic differentiation formula for complex-valued SVD, with the key innovation being the identification of a new gradient term $ar{oldsymbol{A}}_d = rac{1}{2} U S^{-1} (oldsymbol{L}^ op - oldsymbol{L}) V^ op$ that arises due to complex analytic structure.
• This new term has no counterpart in real SVD and is essential for correct gradient computation when the loss function depends on both $U$ and $V$ matrices in a gauge-nontrivial way.
• The derivation shows that the gradient is invariant under phase gauge transformations, and the consistency of the formula is enforced through a gauge invariance condition on the auxiliary matrix $ar{U}^T U + ar{V}^T V - ext{c.c.}$.
• The formula is generalizable to rectangular matrices by adding two extra terms beyond the square case, which account for the non-orthogonal complement of the SVD subspace.
• The term $ar{oldsymbol{A}}_d$ vanishes if the loss function depends only on $U$ or only on $V$, emphasizing that a joint dependence on both is required to activate the new term.
• The derived formula enables accurate and efficient backpropagation in complex tensor networks, supporting applications in quantum many-body physics and differentiable machine learning with complex-valued models.