[Paper Review] On the Equivalence of Automatic and Symbolic Differentiation

Soeren Laue|arXiv (Cornell University)|Apr 5, 2019

Neural Networks and Applications4 citations

TL;DR

This paper demonstrates that reverse mode automatic differentiation and symbolic differentiation are algorithmically equivalent, performing identical operations when computing derivatives. The key insight is that both methods compute the same derivative expressions when common subexpressions are used, debunking the widely held belief that automatic differentiation avoids 'expression swell'—a phenomenon actually caused by poor representation, not differentiation itself.

ABSTRACT

We show that reverse mode automatic differentiation and symbolic differentiation are equivalent in the sense that they both perform the same operations when computing derivatives. This is in stark contrast to the common claim that they are substantially different. The difference is often illustrated by claiming that symbolic differentiation suffers from "expression swell" whereas automatic differentiation does not. Here, we show that this statement is not true. "Expression swell" refers to the phenomenon of a much larger representation of the derivative as opposed to the representation of the original function.

Motivation & Objective

• To challenge the common belief that automatic differentiation and symbolic differentiation are fundamentally different.
• To clarify that 'expression swell'—often cited as a key advantage of automatic differentiation—is not inherent to symbolic differentiation but results from inefficient expression representation.
• To demonstrate that both methods perform identical operations when common subexpressions are properly managed.
• To resolve confusion in the literature about tools like Theano, which are ambiguously classified as using either method.
• To show that control structures do not fundamentally distinguish the two approaches, as both operate on the underlying computational graph.

Proposed method

• The paper models functions using directed acyclic graphs (DAGs), which represent the computational trace of a program.
• It compares reverse mode automatic differentiation and symbolic differentiation by analyzing their differentiation rules on DAGs and expression forests with common subexpressions.
• It formalizes the differentiation process using recursive rules: for each node, the derivative is computed based on its children, using the chain rule.
• It shows that when common subexpressions are stored as pointers (not duplicated), symbolic differentiation and reverse mode automatic differentiation produce identical intermediate and final results.
• It uses Speelpenning’s example (f(x₁,…,xₙ) = x₁x₂…xₙ) to illustrate that both methods compute the same gradient, even though the final output may appear large.
• It argues that expression simplification and common subexpression elimination can be performed during differentiation, not after, making both methods equally efficient.

Experimental results

Research questions

• RQ1 Are reverse mode automatic differentiation and symbolic differentiation truly different in their computational operations?
• RQ2 Does 'expression swell' arise from the differentiation process itself or from the choice of expression representation?
• RQ3 Can symbolic differentiation achieve the same efficiency as automatic differentiation when common subexpressions are properly managed?
• RQ4 How do control structures like if-else or loops affect the equivalence between the two methods?
• RQ5 Is the distinction between automatic and symbolic differentiation a matter of implementation or of fundamental algorithmic difference?

Key findings

• Reverse mode automatic differentiation and symbolic differentiation perform the same set of operations when common subexpressions are used, proving their algorithmic equivalence.
• The phenomenon of 'expression swell' is not caused by symbolic differentiation but by converting a DAG into a tree without common subexpressions; this is a representation issue, not a computational one.
• When common subexpressions are stored as pointers, the size of the derivative expression remains linear in the size of the original DAG, avoiding exponential growth.
• The claim that common subexpression elimination must be done post-hoc is incorrect; it can be integrated during differentiation, making symbolic differentiation as efficient as automatic differentiation.
• Control structures do not break the equivalence, as both methods operate on the underlying computational graph, which abstracts away control flow.
• Speelpenning’s example does not demonstrate inefficiency in symbolic differentiation; both methods compute the same gradient, and the final output size is reduced via simplification, not by inherent algorithmic advantage.