[Paper Review] Interpretable learning of effective dynamics for multiscale systems

Emmanuel Menier, Sebastian Kaltenbach+3 more|arXiv (Cornell University)|Sep 11, 2023

Model Reduction and Neural NetworksPhysics and Astronomy3 citations

TL;DR

This paper proposes iLED, a novel interpretable framework for learning effective dynamics in multiscale systems by combining Mori-Zwanzig and Koopman operator theory with neural network-based dimensionality reduction. It replaces black-box LSTMs with physically grounded linear dynamics and nonlinear closures, achieving state-of-the-art accuracy while enabling full interpretability in three benchmark multiscale systems.

ABSTRACT

The modeling and simulation of high-dimensional multiscale systems is a critical challenge across all areas of science and engineering. It is broadly believed that even with today's computer advances resolving all spatiotemporal scales described by the governing equations remains a remote target. This realization has prompted intense efforts to develop model order reduction techniques. In recent years, techniques based on deep recurrent neural networks have produced promising results for the modeling and simulation of complex spatiotemporal systems and offer large flexibility in model development as they can incorporate experimental and computational data. However, neural networks lack interpretability, which limits their utility and generalizability across complex systems. Here we propose a novel framework of Interpretable Learning Effective Dynamics (iLED) that offers comparable accuracy to state-of-the-art recurrent neural network-based approaches while providing the added benefit of interpretability. The iLED framework is motivated by Mori-Zwanzig and Koopman operator theory, which justifies the choice of the specific architecture. We demonstrate the effectiveness of the proposed framework in simulations of three benchmark multiscale systems. Our results show that the iLED framework can generate accurate predictions and obtain interpretable dynamics, making it a promising approach for solving high-dimensional multiscale systems.

Motivation & Objective

• Address the critical challenge of simulating high-dimensional multiscale systems where resolving all spatiotemporal scales remains computationally intractable.
• Overcome the lack of interpretability in state-of-the-art deep learning-based model order reduction methods, such as those using LSTMs.
• Develop a framework that maintains high predictive accuracy while providing physically interpretable dynamics through theoretically grounded dynamical models.
• Enable generalizability and trust in complex system simulations by embedding physical principles into the learned dynamics.
• Demonstrate the framework's effectiveness across diverse multiscale systems, including the Kuramoto-Sivashinsky equation and flow around a cylinder at varying Reynolds numbers.

Proposed method

• Employ neural autoencoders to identify low-dimensional, nonlinear manifolds that capture the dominant dynamics of high-dimensional systems.
• Replace the standard LSTM in prior Learning Effective Dynamics (LED) frameworks with a physically motivated dynamical model based on Koopman operator and Mori-Zwanzig formalism.
• Model the latent dynamics using a linear component represented by a skew-symmetric matrix with diagonal damping (Aθ = W − Wᵀ − diag(w), w ∈ ℝ₊ᵈₕ), ensuring energy-conserving and stable evolution.
• Incorporate a nonlinear closure term via a deep neural network (Ψ₂) to account for unresolved scale interactions, enhancing accuracy beyond linear dynamics.
• Use SiLU activation functions and batch normalization in the autoencoder and dynamics networks to stabilize training and improve generalization.
• Enforce latent space centering via a dedicated loss term (ℒ_centering) to maintain stability and interpretability, especially in low batch-size settings.

Experimental results

Research questions

• RQ1 Can a deep learning-based model order reduction framework achieve comparable accuracy to black-box LSTMs while providing full interpretability?
• RQ2 To what extent can the Mori-Zwanzig and Koopman operator theories be leveraged to construct a theoretically grounded, interpretable dynamical model for multiscale systems?
• RQ3 How does the inclusion of a physically motivated nonlinear closure term improve predictive accuracy compared to purely linear latent dynamics?
• RQ4 Can the iLED framework maintain high accuracy and interpretability across diverse multiscale systems with varying complexity and scale separation?
• RQ5 What is the impact of latent space centering and architectural choices on the stability and performance of the learned dynamics?

Key findings

• The iLED framework achieves state-of-the-art prediction accuracy on the Kuramoto-Sivashinsky (KS) equation, matching or exceeding that of LSTM-based LED models.
• For the flow around a cylinder at Re=100 and Re=750, iLED accurately captures the complex spatiotemporal dynamics using latent dimensions of dₕ=3 and dₕ=16, respectively.
• The use of a skew-symmetric matrix with diagonal damping (Aθ) ensures stable, energy-conserving evolution of the latent dynamics, enabling long-term prediction fidelity.
• The inclusion of a nonlinear closure term (Ψ₂) significantly improves model accuracy over linear-only dynamics, particularly in systems with strong nonlinearities.
• The ℒ_centering loss effectively maintains a centered latent space, which is critical for the stability and interpretability of the linear dynamics component.
• The framework enables interpretable dynamics by explicitly modeling the linear evolution and closure terms, allowing direct physical interpretation of the learned system behavior.