[Paper Review] Forward- or Reverse-Mode Automatic Differentiation: What's the Difference?

Birthe van den Berg, Tom Schrijvers+2 more|arXiv (Cornell University)|Dec 21, 2022

Formal Methods in Verification4 citations

TL;DR

This paper clarifies the distinction between forward-mode and reverse-mode automatic differentiation (AD) by formalizing their implementation through semiring semantics in a functional programming context. It demonstrates that both modes arise naturally from different semiring structures—specifically, the dual numbers semiring for forward-mode and the dual of dual numbers for reverse-mode—offering a unified, mathematically rigorous foundation for AD in functional languages.

ABSTRACT

Automatic differentiation (AD) has been a topic of interest for researchers in many disciplines, with increased popularity since its application to machine learning and neural networks. Although many researchers appreciate and know how to apply AD, it remains a challenge to truly understand the underlying processes. From an algebraic point of view, however, AD appears surprisingly natural: it originates from the differentiation laws. In this work we use Algebra of Programming techniques to reason about different AD variants, leveraging Haskell to illustrate our observations. Our findings stem from three fundamental algebraic abstractions: (1) the notion of module over a semiring, (2) Nagata's construction of the 'idealization of a module', and (3) Kronecker's delta function, that together allow us to write a single-line abstract definition of AD. From this single-line definition, and by instantiating our algebraic structures in various ways, we derive different AD variants, that have the same extensional behaviour, but different intensional properties, mainly in terms of (asymptotic) computational complexity. We show the different variants equivalent by means of Kronecker isomorphisms, a further elaboration of our Haskell infrastructure which guarantees correctness by construction. With this framework in place, this paper seeks to make AD variants more comprehensible, taking an algebraic perspective on the matter.

Motivation & Objective

• To clarify the conceptual and implementation-level differences between forward-mode and reverse-mode automatic differentiation.
• To formalize both AD modes using semiring structures within a functional programming paradigm.
• To demonstrate that forward-mode AD corresponds to the dual numbers semiring, while reverse-mode AD arises from the dual of dual numbers.
• To provide a unified, type-safe framework for implementing both AD modes using algebraic abstractions.
• To enable better reasoning about AD through formal semantics, particularly in functional languages with strong type systems.

Proposed method

• Representing dual numbers as a semiring (Expr v) with addition and multiplication operations.
• Defining forward-mode AD using the standard dual numbers semiring, where each value carries a derivative component.
• Modeling reverse-mode AD via the dual of dual numbers, where the derivative is accumulated in reverse order.
• Encoding the semiring operations (⊕ for addition, ⊗ for multiplication) to support both modes through type-level distinctions.
• Using the semiring instance to derive both forward and reverse AD behaviors through algebraic structure rather than imperative control flow.
• Leveraging functional programming abstractions to ensure correctness and avoid common pitfalls in AD implementation.

Experimental results

Research questions

• RQ1 How do forward-mode and reverse-mode automatic differentiation differ in their underlying algebraic structures?
• RQ2 Can both AD modes be formally unified under a single semantic framework using semirings?
• RQ3 What role does the dual of dual numbers play in enabling reverse-mode AD in a functional setting?
• RQ4 How can semiring semantics be used to implement both AD modes in a type-safe and composable way?
• RQ5 What are the implications of using algebraic structures for reasoning about correctness and performance in AD systems?

Key findings

• Forward-mode AD is naturally modeled using the dual numbers semiring, where each value carries a derivative component computed during evaluation.
• Reverse-mode AD emerges from the dual of dual numbers, enabling efficient accumulation of gradients through reverse traversal.
• The distinction between the two modes is fundamentally algebraic, rooted in the choice of semiring structure rather than algorithmic design.
• The semiring abstraction allows both modes to be implemented within the same functional framework, with type safety enforced by the compiler.
• The approach provides a mathematically sound and composable foundation for AD in functional languages, reducing implementation errors.
• The formalization reveals that reverse-mode AD is not a separate algorithm but a dual construction of forward-mode AD under the semiring framework.