[Paper Review] Dynamical Systems and Neural Networks

Akshunna S. Dogra|arXiv (Cornell University)|Apr 20, 2020

Model Reduction and Neural Networks5 references4 citations

TL;DR

This paper proposes a Koopman Operator Theory (KOT)-driven framework to accelerate training of neural network (NN) solvers for dynamical systems by leveraging spectral properties of Koopman operators. By identifying Koopman eigenfunctions and modes from data, the method reduces computational costs and enables efficient, unsupervised training while preserving accuracy and differentiability of NN-based functional approximations for system evolution.

ABSTRACT

Neural Networks (NNs) have been identified as a potentially powerful tool in the study of complex dynamical systems. A good example is the NN differential equation (DE) solver, which provides closed form, differentiable, functional approximations for the evolution of a wide variety of dynamical systems. A major disadvantage of such NN solvers can be the amount of computational resources needed to achieve accuracy comparable to existing numerical solvers. We present new strategies for existing dynamical system NN DE solvers, making efficient use of the extit{learnt} information, to speed up their training process, while still pursuing a completely unsupervised approach. We establish a fundamental connection between NN theory and dynamical systems theory via Koopman Operator Theory (KOT), by showing that the usual training processes for Neural Nets are fertile ground for identifying multiple Koopman operators of interest. We end by illuminating certain applications that KOT might have for NNs in general.

Motivation & Objective

• To reduce the high computational cost of training neural network solvers for dynamical systems while maintaining accuracy.
• To establish a fundamental link between neural network training and Koopman Operator Theory (KOT).
• To enable efficient, unsupervised training of NNs by exploiting Koopman spectral information for error correction and convergence acceleration.
• To demonstrate that Koopman-based training strategies can outperform standard gradient-based methods in terms of computational efficiency.
• To explore the use of Koopman modes and eigenfunctions as tools for identifying and correcting prediction errors in NN-based dynamical system solvers.

Proposed method

• Uses Koopman Operator Theory (KOT) to model the evolution of neural network weights and loss functions as a linear dynamical system in a lifted space.
• Identifies Koopman eigenfunctions and eigenvalues from data to approximate the dynamics of the loss and weight updates, enabling spectral-based optimization.
• Applies Koopman mode decomposition (KMD) to extract dominant modes that govern the training dynamics, reducing reliance on iterative gradient updates.
• Integrates Koopman-based error correction into the loss function by modeling residual errors as Koopman-mode expansions.
• Employs data-driven Koopman operator approximations ($\hat{U}$) to replace or augment traditional backpropagation, minimizing intermediate computational overhead.
• Utilizes the structure of the Koopman operator to enable parallel, non-causal evaluation of loss terms, similar to the original NN solver’s unsupervised training.

Experimental results

Research questions

• RQ1 Can Koopman Operator Theory be used to model and accelerate the training dynamics of neural networks for dynamical systems?
• RQ2 How can spectral information from Koopman operators improve convergence and reduce computational cost in unsupervised neural network training?
• RQ3 To what extent can Koopman modes and eigenfunctions be used to predict and correct errors in neural network predictions for dynamical systems?
• RQ4 Can data-driven Koopman operators replace or enhance standard gradient descent in training neural network solvers?
• RQ5 What is the relationship between Koopman operator dynamics and the convergence behavior of neural network weights and loss functions?

Key findings

• The training process of neural networks for dynamical systems can be modeled as a Koopman dynamical system, enabling spectral analysis of learning dynamics.
• Koopman eigenfunctions and modes can be used to approximate the evolution of loss and weights, reducing reliance on iterative gradient updates.
• Data-driven Koopman operator approximations allow for efficient, unsupervised training with computational complexity comparable to matrix multiplication, avoiding expensive derivative calculations.
• The method enables parallel evaluation of loss terms, eliminating sequential dependency and accelerating training.
• Koopman-based error correction improves prediction accuracy and reduces training time, especially in late-stage convergence where gradient descent slows.
• The framework reveals that Koopman operators can identify limit points and invariant structures in both loss and weight spaces, providing deeper insight into training stability.