[Paper Review] Deep Differential System Stability -- Learning advanced computations from examples

François Charton, Amaury Hayat+1 more|arXiv (Cornell University)|Jun 11, 2020

Model Reduction and Neural Networks20 references4 citations

TL;DR

This paper demonstrates that transformers trained on large synthetic datasets can learn to predict advanced qualitative and quantitative properties of differential systems—such as local stability, behavior at infinity, and controllability—achieving near-perfect estimates of qualitative features and strong approximations of numerical values, showing neural networks can internalize complex mathematical theorems without explicit symbolic knowledge.

ABSTRACT

Can advanced mathematical computations be learned from examples? Using transformers over large generated datasets, we train models to learn properties of differential systems, such as local stability, behavior at infinity and controllability. We achieve near perfect estimates of qualitative characteristics of the systems, and good approximations of numerical quantities, demonstrating that neural networks can learn advanced theorems and complex computations without built-in mathematical knowledge.

Motivation & Objective

• To investigate whether neural networks can learn advanced mathematical computations, such as stability and controllability of differential systems, from purely data-driven examples.
• To assess whether transformers can infer qualitative properties of differential systems without explicit symbolic or analytical training.
• To evaluate the performance of neural networks in approximating complex numerical quantities related to differential systems using only generated training data.
• To explore the extent to which deep learning models can internalize and generalize mathematical theorems through pattern recognition in synthetic datasets.

Proposed method

• Training transformer-based models on large-scale synthetic datasets of differential systems with known properties.
• Generating diverse differential systems with labeled qualitative and quantitative characteristics, including local stability, asymptotic behavior, and controllability.
• Using attention mechanisms in transformers to capture long-range dependencies and structural patterns in system dynamics.
• Formulating the prediction task as a sequence-to-sequence or classification-regression hybrid, depending on the property type.
• Evaluating model performance using metrics for both qualitative classification and numerical regression tasks.

Experimental results

Research questions

• RQ1 Can transformers learn to classify the local stability of differential systems from training examples alone?
• RQ2 To what extent can neural networks predict the behavior of systems at infinity without explicit mathematical formulation?
• RQ3 Can models approximate numerical quantities like controllability indices with high accuracy using only data?
• RQ4 How well do models generalize to unseen system structures not present in the training data?

Key findings

• The model achieved near-perfect accuracy in predicting qualitative properties such as local stability and asymptotic behavior of differential systems.
• Numerical approximations of system quantities, such as controllability measures, were consistently accurate, demonstrating strong regression performance.
• The model generalized effectively to novel system types not seen during training, indicating robust pattern learning.
• Performance remained high even when systems had complex nonlinear dynamics, suggesting effective capture of underlying mathematical structure.