[Paper Review] CHAD for Expressive Total Languages

Fernando Lucatelli Nunes, Matthijs Vákár|Data Archiving and Networked Services (DANS)|Oct 1, 2021

Logic, programming, and type systems33 references4 citations

TL;DR

This paper extends Combinatory Homomorphic Automatic Differentiation (CHAD) to expressive total functional programming languages with tuple, sum, inductive, coinductive, and function types. By developing a categorical semantics using $Σ$-types and novel logical relations techniques, it proves CHAD correctly computes mathematical derivatives, enabling principled, purely functional, and formally correct forward and reverse mode AD in rich type systems.

ABSTRACT

We show how to apply forward and reverse mode Combinatory Homomorphic Automatic Differentiation (CHAD) to total functional programming languages with expressive type systems featuring the combination of - tuple types; - sum types; - inductive types; - coinductive types; - function types. We achieve this by analysing the categorical semantics of such types in $Σ$-types (Grothendieck constructions) of suitable categories. Using a novel categorical logical relations technique for such expressive type systems, we give a correctness proof of CHAD in this setting by showing that it computes the usual mathematical derivative of the function that the original program implements. The result is a principled, purely functional and provably correct method for performing forward and reverse mode automatic differentiation (AD) on total functional programming languages with expressive type systems.

Motivation & Objective

• To extend Combinatory Homomorphic Automatic Differentiation (CHAD) to total functional programming languages with expressive type systems, including tuple, sum, inductive, coinductive, and function types.
• To provide a formal, denotational foundation for CHAD in such expressive type systems using categorical semantics.
• To establish correctness of CHAD by proving it computes the standard mathematical derivative using novel categorical logical relations techniques.
• To enable principled, purely functional, and formally verified source-code transformation for forward and reverse mode automatic differentiation in higher-order, richly-typed languages.
• To support practical implementation of AD in languages with complex data structures like streams and recursive trees, while maintaining efficiency and correctness.

Proposed method

• Develops a categorical semantics for expressive type systems using $Σ$-types (Grothendieck constructions) over suitable base categories.
• Defines a target language for CHAD that supports the full range of expressive types, enabling homomorphic encoding of primal and tangent computations.
• Derives forward and reverse CHAD algorithms as uniquely defined homomorphic functors from the source language to the target language.
• Introduces a novel categorical logical relations technique (sconing) for reasoning about programs involving both inductive and coinductive types.
• Applies the logical relations construction over a concrete denotational semantics to prove that CHAD correctly computes derivatives.
• Uses initial algebra and coalgebra semantics to model recursive and corecursive data types, ensuring well-defined behavior under differentiation.

Experimental results

Research questions

• RQ1 How can CHAD be extended to support inductive and coinductive types in total functional languages?
• RQ2 What categorical framework supports the semantics of expressive type systems (including sums, products, and function types) in a way that enables compositional AD?
• RQ3 Can a logical relations technique be developed that simultaneously handles both inductive and coinductive types in the context of automatic differentiation?
• RQ4 Does CHAD correctly compute the standard mathematical derivative for functions operating on complex data structures like streams and trees?
• RQ5 How can the correctness of CHAD be formally proven in a setting with higher-order functions and rich type constructs?

Key findings

• CHAD is successfully extended to total functional languages with tuple, sum, inductive, coinductive, and function types, enabling both forward and reverse mode AD.
• The paper establishes a formal correctness proof of CHAD using a novel categorical logical relations technique that handles both inductive and coinductive types.
• The derived CHAD transformation is shown to compute the correct mathematical derivative for functions between inductive types, including recursive data structures.
• The method is purely functional and based on homomorphic functors, ensuring compositional and type-safe differentiation.
• The approach supports efficient, static source-code transformation without runtime tracing or custom operational semantics.
• The correctness result holds even for programs involving infinite or lazy data structures, thanks to the use of coalgebraic semantics and coinductive reasoning.