[Paper Review] Principles of Automatic Differentiation

Philipp Hoffmann|arXiv (Cornell University)|Nov 3, 2014

Numerical methods for differential equations7 references3 citations

TL;DR

This paper establishes the mathematical foundations of Automatic Differentiation (AD), unifying multiple formulations of Forward Mode AD—such as dual numbers, matrix-vector multiplication, Taylor expansions, and push-forward operators—by demonstrating their computational equivalence. It further introduces higher-order Forward AD and contrasts Reverse Mode AD with Forward Mode, clarifying their shared computational roots.

ABSTRACT

This article provides an overview of some of the mathematical principles of Automatic Differentiation (AD). In particular, we summarise different descriptions of the Forward Mode of AD, like the matrix-vector product based approach, the idea of lifting functions to the algebra of dual numbers, the method of Taylor series expansion on dual numbers and the application of the push-forward operator, and explain why they all reduce to the same actual chain of computations. We further give a short mathematical description of some methods of higher-order Forward AD and, at the end of this paper, describe the Reverse Mode and explain its relation to the Forward Mode.

Motivation & Objective

• To unify diverse mathematical formulations of Forward Mode Automatic Differentiation into a coherent computational framework.
• To clarify the equivalence of seemingly different AD approaches, such as dual numbers and push-forward operators, through shared computational chains.
• To extend the analysis to higher-order Forward AD and to relate Reverse Mode AD to Forward Mode in terms of underlying principles.
• To provide a mathematically rigorous foundation for AD that supports implementation and theoretical understanding.

Proposed method

• Formalizing Forward Mode AD using the algebra of dual numbers, where each number carries a value and a derivative component.
• Expressing the chain rule via matrix-vector multiplication in the context of directional derivatives.
• Deriving the same computational results through Taylor series expansions on dual numbers, preserving order of derivatives.
• Applying the push-forward operator to model the propagation of derivatives through composite functions.
• Extending the framework to higher-order derivatives using recursive lifting and higher-order dual numbers.
• Comparing Reverse Mode AD to Forward Mode by analyzing adjoint propagation and its duality with forward accumulation.

Experimental results

Research questions

• RQ1 How do different formulations of Forward Mode AD—dual numbers, matrix-vector products, Taylor series, and push-forward—yield identical computational results?
• RQ2 What mathematical structure underlies the equivalence of these diverse AD formulations?
• RQ3 How can higher-order derivatives be systematically computed using Forward Mode AD principles?
• RQ4 In what way is Reverse Mode AD related to Forward Mode in terms of computational and algebraic structure?

Key findings

• All formulations of Forward Mode AD—dual numbers, matrix-vector products, Taylor series, and push-forward—compute the same chain of derivatives due to isomorphic computational behavior.
• The use of dual numbers provides an algebraic framework that naturally encodes the chain rule for first-order derivatives.
• Higher-order Forward AD can be constructed by extending dual numbers to include higher-order terms, enabling systematic derivative computation.
• Reverse Mode AD is mathematically dual to Forward Mode, with the key difference lying in the direction of derivative propagation and the use of adjoints.