[Paper Review] Neural Integration of Continuous Dynamics

Margaret Trautner, Sai Ravela|arXiv (Cornell University)|Nov 23, 2019

Model Reduction and Neural Networks2 references4 citations

TL;DR

This paper introduces a compact, recurrent discrete-time neural network (DtNN) that performs neural integration of continuous-time dynamical systems using standard numerical integrators—explicit Runge-Kutta and semi-implicit Adams-Bashforth-Moulton methods—achieving exact equivalence to numerical integration when the system is polynomial, as demonstrated on the Lorenz-63 model.

ABSTRACT

Neural dynamical systems are dynamical systems that are described at least in part by neural networks. The class of continuous-time neural dynamical systems must, however, be numerically integrated for simulation and learning. Here, we present a compact neural circuit for two common numerical integrators: the explicit fixed-step Runge-Kutta method of any order and the semi-implicit/predictor-corrector Adams-Bashforth-Moulton method. Modeled as constant-sized recurrent networks embedding a continuous neural differential equation, they achieve fully neural temporal output. Using the polynomial class of dynamical systems, we demonstrate the equivalence of neural and numerical integration.

Motivation & Objective

• To develop a fully neural, recurrent discrete-time network that performs numerical integration of continuous dynamical systems without relying on external numerical solvers.
• To demonstrate that neural integration via DtNNs matches the accuracy of classical numerical integration for polynomial dynamical systems.
• To show that the network architecture remains compact and invariant to integration order, unlike prior approaches that scale with order.
• To establish a foundation for theory-guided learning in systems dynamics by embedding governing equations into PolyNet-based neural circuits.
• To enable end-to-end differentiable simulation and training of neural ODEs using purely neural computation, supporting applications in environmental modeling and adaptive systems.

Proposed method

• Constructs a recurrent DtNN that embeds the derivative computation and numerical stepping of explicit Runge-Kutta methods into a single neural circuit.
• Uses multiplicative nodes to enable order-invariant architecture: network size does not increase with integration order.
• Applies the same methodology to semi-implicit methods by coupling a two-step Adams-Bashforth predictor with a two-step Adams-Moulton corrector in a single recurrent network.
• Employs PolyNets—continuous-time neural networks with polynomial nonlinearities—as the underlying dynamical system representation for exact integration.
• Defines the neural integration process as a sequence of discrete-time outputs generated by the recurrent network, each representing a time step of the solution.
• Validates correctness by showing convergence between neural and numerical integration up to machine precision for the Lorenz-63 system with known polynomial coefficients.

Experimental results

Research questions

• RQ1 Can a single recurrent discrete-time neural network perform numerical integration of continuous dynamical systems with the same accuracy as classical solvers?
• RQ2 Does the proposed neural integration method maintain fixed network size regardless of integration order, unlike prior neural circuit implementations?
• RQ3 To what extent can PolyNets enable exact neural integration for systems governed by polynomial differential equations?
• RQ4 How does neural integration compare to standard numerical integration in terms of convergence and numerical representation for chaotic systems like Lorenz-63?
• RQ5 Can neural integration be used as a foundation for theory-guided learning in systems dynamics, where governing equations are known but parameters are learned from data?

Key findings

• The proposed recurrent DtNN performs exact neural integration of continuous-time dynamical systems when the system is polynomial, achieving equivalence to classical numerical integration.
• For the Lorenz-63 system, neural integration via PolyNets converges to numerical integration up to machine precision, demonstrating numerical correctness.
• The network size remains constant regardless of the order of the Runge-Kutta or Adams-Bashforth-Moulton methods, enabling scalable and compact implementation.
• PolyNets require only two nodes to exactly represent the Lorenz-63 system, whereas standard feed-forward networks with tanh activation require at least six nodes, highlighting efficiency.
• The method enables end-to-end differentiable simulation of neural ODEs using only neural computation, eliminating the need for external numerical integrators.
• The approach supports theory-guided learning by embedding known governing equations into the network structure, reducing data and parameter requirements compared to black-box neural networks.