[Paper Review] GN-SINDy: Greedy Sampling Neural Network in Sparse Identification of Nonlinear Partial Differential Equations

Ali Forootani, Kapadia, Harshit+3 more|arXiv (Cornell University)|May 14, 2024

Model Reduction and Neural Networks4 citations

TL;DR

This paper introduces GN-SINDy, a novel method that integrates greedy sampling with deep neural networks (DNNs) and sparse identification (SINDy) to improve data efficiency and accuracy in discovering nonlinear partial differential equations (PDEs). By using the Discrete Empirical Interpolation Method (DEIM) to select informative snapshots and applying STRidge for sparsity promotion, GN-SINDy achieves high-accuracy PDE discovery using only 0.5–0.874% of the full dataset, outperforming prior methods like DeePyMoD.

ABSTRACT

The sparse identification of nonlinear dynamical systems (SINDy) is a data-driven technique employed for uncovering and representing the fundamental dynamics of intricate systems based on observational data. However, a primary obstacle in the discovery of models for nonlinear partial differential equations (PDEs) lies in addressing the challenges posed by the curse of dimensionality and large datasets. Consequently, the strategic selection of the most informative samples within a given dataset plays a crucial role in reducing computational costs and enhancing the effectiveness of SINDy-based algorithms. To this aim, we employ a greedy sampling approach to the snapshot matrix of a PDE to obtain its valuable samples, which are suitable to train a deep neural network (DNN) in a SINDy framework. SINDy based algorithms often consist of a data collection unit, constructing a dictionary of basis functions, computing the time derivative, and solving a sparse identification problem which ends to regularised least squares minimization. In this paper, we extend the results of a SINDy based deep learning model discovery (DeePyMoD) approach by integrating greedy sampling technique in its data collection unit and new sparsity promoting algorithms in the least squares minimization unit. In this regard we introduce the greedy sampling neural network in sparse identification of nonlinear partial differential equations (GN-SINDy) which blends a greedy sampling method, the DNN, and the SINDy algorithm. In the implementation phase, to show the effectiveness of GN-SINDy, we compare its results with DeePyMoD by using a Python package that is prepared for this purpose on numerous PDE discovery

Motivation & Objective

• To address the curse of dimensionality and high computational cost in sparse identification of nonlinear PDEs using large datasets.
• To improve data efficiency in SINDy-based PDE discovery by replacing random sampling with a greedy sampling strategy that selects the most informative snapshots.
• To enhance model discovery accuracy by integrating advanced sparsity-promoting solvers (e.g., STRidge) into the SINDy framework.
• To extend the DeePyMoD framework by replacing its random sampling and default solvers with a principled greedy sampling and optimized sparse regression pipeline.
• To demonstrate that reduced data size and improved sampling strategy lead to superior PDE recovery with minimal hyperparameter tuning.

Proposed method

• Employ the Discrete Empirical Interpolation Method (DEIM) to identify the most informative snapshot samples from the PDE's solution matrix, forming a greedy sampling strategy.
• Use a deep neural network (DNN) to learn the underlying dynamics from the selected greedy samples, enabling accurate approximation of spatial and temporal derivatives.
• Integrate the DNN predictions into the SINDy framework by constructing a dictionary of candidate basis functions (e.g., u, ux, uxx, u^2ux, etc.).
• Apply a sequential threshold ridge regression (STRidge) algorithm to enforce sparsity and promote correct term selection during the sparse identification step.
• Use a greedy sampling threshold (ε_thr = 10⁻⁵) and time division (t_div = 2) to control sample selection and temporal resolution in the training process.
• Compare the performance of GN-SINDy against DeePyMoD using identical DNN architectures and evaluation metrics across multiple PDEs (Burgers’, Allen-Cahn, KdV).

Experimental results

Research questions

• RQ1 Can greedy sampling of PDE snapshots significantly reduce data requirements for accurate SINDy-based PDE discovery compared to random sampling?
• RQ2 How does integrating STRidge sparsity promotion improve the accuracy and robustness of PDE model discovery in the presence of noise and high-dimensional data?
• RQ3 To what extent does the combination of DEIM-based sampling and DNN-based regression enhance the recovery of complex nonlinear PDEs like KdV and Allen-Cahn?
• RQ4 How sensitive is GN-SINDy’s performance to hyperparameters such as DNN architecture and sampling strategy, and can it generalize across diverse PDE classes?
• RQ5 Does GN-SINDy outperform state-of-the-art methods like DeePyMoD in terms of data efficiency and model accuracy for nonlinear PDE identification?

Key findings

• GN-SINDy successfully recovers the Korteweg-de Vries (KdV) equation using only 0.874% of the full dataset (900 out of ~103,000 snapshots), demonstrating high data efficiency.
• The method achieves exact recovery of the KdV equation with a DNN architecture of [2, 64, 64, 64, 64, 1], yielding the estimated PDE: u_t - 5.9812 u u_x - 0.9923 u_xxx = 0.
• GN-SINDy outperforms DeePyMoD in all tested PDEs: it recovers Burgers’ equation with 0.5% of the data, Allen-Cahn with 0.1%, and KdV with 0.874% of the data.
• The coefficient evolution in GN-SINDy shows faster convergence for the u_xxx term compared to the u u_x term, indicating improved learning dynamics due to better sample selection.
• When using a DNN with 8 or 16 neurons per layer, GN-SINDy fails to recover the KdV equation, highlighting the sensitivity of model discovery to DNN architecture and the importance of proper hyperparameter tuning.
• DeePyMoD, using random sampling and default LASSO/OLS solvers, produces a highly inaccurate and overfit model for the KdV equation, including spurious terms like u^2 u_xxx and u^2 u_x, while GN-SINDy recovers the correct sparse structure.