[Paper Review] FINDE: Neural Differential Equations for Finding and Preserving Invariant Quantities

Takashi Matsubara, Takaharu Yaguchi|arXiv (Cornell University)|Oct 1, 2022

Model Reduction and Neural Networks4 citations

TL;DR

FINDE introduces a neural differential equation framework that discovers and preserves unknown first integrals (invariant quantities) in dynamical systems from data, using projection and discrete gradient methods. It enables long-term accurate prediction and scientific discovery by exactly conserving first integrals in discrete time, outperforming prior methods in both stability and generalization across diverse systems.

ABSTRACT

Many real-world dynamical systems are associated with first integrals (a.k.a. invariant quantities), which are quantities that remain unchanged over time. The discovery and understanding of first integrals are fundamental and important topics both in the natural sciences and in industrial applications. First integrals arise from the conservation laws of system energy, momentum, and mass, and from constraints on states; these are typically related to specific geometric structures of the governing equations. Existing neural networks designed to ensure such first integrals have shown excellent accuracy in modeling from data. However, these models incorporate the underlying structures, and in most situations where neural networks learn unknown systems, these structures are also unknown. This limitation needs to be overcome for scientific discovery and modeling of unknown systems. To this end, we propose first integral-preserving neural differential equation (FINDE). By leveraging the projection method and the discrete gradient method, FINDE finds and preserves first integrals from data, even in the absence of prior knowledge about underlying structures. Experimental results demonstrate that FINDE can predict future states of target systems much longer and find various quantities consistent with well-known first integrals in a unified manner.

Motivation & Objective

• To address the limitation of existing neural networks that require prior knowledge of first integrals for accurate modeling of dynamical systems.
• To develop a unified framework capable of discovering unknown first integrals from data, even when underlying geometric structures are unknown.
• To ensure exact conservation of first integrals in discrete-time simulations, improving prediction accuracy and stability.
• To combine with existing physics-informed models, allowing integration of known first integrals without redundancy.
• To enable scientific discovery by identifying both known and novel invariant quantities in complex systems.

Proposed method

• FINDE employs a continuous-time version using projection methods to constrain the dynamics to a submanifold where first integrals are preserved.
• The discrete-time version leverages the discrete gradient method to ensure exact conservation of first integrals up to numerical precision.
• It formulates the learning objective as a constrained optimization problem, enforcing the gradient of the first integral to be zero in discrete time.
• The method is trained end-to-end on observed trajectories, automatically identifying the number and form of first integrals through a latent space parameterization.
• It integrates with neural ODEs by modifying the vector field to respect the invariant structure without requiring symbolic knowledge of the system.
• The framework is compatible with existing architectures like HNNs and can be extended to include known first integrals as constraints.

Experimental results

Research questions

• RQ1 Can a neural network discover unknown first integrals in dynamical systems from purely observational data?
• RQ2 How can first integrals be preserved exactly in discrete-time simulations without prior knowledge of the system's geometric structure?
• RQ3 Can the proposed method outperform existing physics-informed neural networks in long-term prediction accuracy?
• RQ4 How does the framework handle first integrals that vary across different initial conditions, such as energy or momentum?
• RQ5 Can the method be combined with existing invariant-preserving models to improve generalization and avoid redundant learning?

Key findings

• FINDE achieved significantly better long-term prediction performance than standard NODEs, with a 1-step error of 0.63 ± 0.02 and VPT of 0.994 ± 0.002 on the 2-pend dataset with K=3.
• The discrete-time version (dFINDE) preserved first integrals exactly up to rounding errors, while standard NODEs and modified NPM failed to maintain invariants.
• On the 2-pend dataset, dFINDE achieved 100% success rate across 5 trials, while the modified NPM failed in 50% of trials due to optimization divergence or underflow.
• For K=6, NODE with cFINDE failed catastrophically on the training set (1-step error: 9.93 ± 0.00), indicating that incorrect submanifold dimensionality severely degrades performance.
• The method successfully discovered first integrals in energy-dissipating systems beyond just energy, demonstrating its capacity for scientific discovery.
• dFINDE outperformed the modified NPM in both accuracy and robustness, with lower error and higher success rate, even when the NPM was adapted to handle time-varying first integrals.