[Paper Review] The differential calculus of causal functions

David Sprunger, Bart Jacobs|arXiv (Cornell University)|Apr 24, 2019

Neural Networks and ApplicationsComputer Science10 references3 citations

TL;DR

This paper introduces a novel differential calculus for causal functions—functions that map infinite sequences to infinite sequences in a time-ordered, memoryless manner—by defining a unique infinite-dimensional derivative that avoids normed vector spaces. The key contribution is a recurrence rule enabling exact, compositional differentiation of recurrent neural networks like Elman networks, validated through manual computation and gradient-based training with accurate local approximations.

ABSTRACT

Causal functions of sequences occur throughout computer science, from theory to hardware to machine learning. Mealy machines, synchronous digital circuits, signal flow graphs, and recurrent neural networks all have behaviour that can be described by causal functions. In this work, we examine a differential calculus of causal functions which includes many of the familiar properties of standard multivariable differential calculus. These causal functions operate on infinite sequences, but this work gives a different notion of an infinite-dimensional derivative than either the Fréchet or Gateaux derivative used in functional analysis. In addition to showing many standard properties of differentiation, we show causal differentiation obeys a unique recurrence rule. We use this recurrence rule to compute the derivative of a simple recurrent neural network called an Elman network by hand and describe how the computed derivative can be used to train the network.

Motivation & Objective

• To develop a rigorous differential calculus for causal functions operating on infinite sequences, avoiding reliance on Fréchet or Gateaux derivatives.
• To address the lack of compositional rules (e.g., chain rule, sum rule) in existing backpropagation-through-time methods for recurrent networks.
• To provide a mathematically sound, finite-dimensional-like derivative notion for infinite-dimensional causal systems, enabling practical gradient computation.
• To demonstrate the utility of the calculus in training Elman networks via exact derivative computation and parameter updates.

Proposed method

• Define causal functions as those where the k-th output depends only on the first k inputs, formalized via a slicing-based condition.
• Introduce a new notion of derivative for causal functions on real-vector sequences, based on finite approximants rather than normed spaces.
• Establish classical calculus rules (sum, product, chain, etc.) adapted to causal functions, using recursive structure and finite approximations.
• Derive a novel recurrence rule (Theorem 4.16) that enables differentiation of recursive systems by propagating derivatives backward through time.
• Apply the calculus to an Elman network by hand, computing the derivative of the network’s output with respect to a weight parameter.
• Use the computed derivative to guide parameter updates in a gradient-descent fashion, demonstrating convergence toward desired behavior.

Experimental results

Research questions

• RQ1 Can a differential calculus be defined for causal functions on infinite sequences that avoids the use of normed vector spaces and standard functional analytic derivatives?
• RQ2 What compositional rules—such as chain, sum, or product rules—hold in this new calculus, and how do they differ from classical multivariable calculus?
• RQ3 Is there a unique recurrence rule that enables efficient and exact differentiation of recursive systems like recurrent neural networks?
• RQ4 How can this calculus be applied to train a real-world recurrent network architecture such as the Elman network?
• RQ5 To what extent does the derivative provide an accurate local approximation for parameter updates in non-linear, infinite-dimensional systems?

Key findings

• The proposed derivative is distinct from Fréchet and Gateaux derivatives, as it is defined via finite approximants rather than limits in normed spaces.
• The calculus supports standard rules such as sum, product, chain, and reciprocal rules, all adapted to the causal setting.
• A novel recurrence rule (Theorem 4.16) is derived, which uniquely enables the differentiation of recursive systems by propagating dual numbers through time.
• Manual differentiation of an Elman network using the calculus yields a derivative that predicts parameter updates with ~5% error for small steps, validating its accuracy.
• Larger parameter adjustments guided by the derivative lead to greater performance gains but also higher error (25–65% overshot), confirming the local validity of the linear approximation.
• The calculus enables exact, compositional gradient computation in recurrent networks without unrolling, offering a theoretical foundation for backpropagation-through-time.