[Paper Review] DR-RNN: A deep residual recurrent neural network for model reduction

J. Nagoor Kani, Ahmed H. Elsheikh|arXiv (Cornell University)|Sep 4, 2017

Model Reduction and Neural Networks19 references71 citations

TL;DR

The paper proposes DR-RNN, a deep residual recurrent neural network that minimizes residuals of discretized dynamical systems and integrates with POD/DEIM for efficient reduced order modeling of nonlinear ODEs/PDEs.

ABSTRACT

We introduce a deep residual recurrent neural network (DR-RNN) as an efficient model reduction technique for nonlinear dynamical systems. The developed DR-RNN is inspired by the iterative steps of line search methods in finding the residual minimiser of numerically discretized differential equations. We formulate this iterative scheme as stacked recurrent neural network (RNN) embedded with the dynamical structure of the emulated differential equations. Numerical examples demonstrate that DR-RNN can effectively emulate the full order models of nonlinear physical systems with a significantly lower number of parameters in comparison to standard RNN architectures. Further, we combined DR-RNN with Proper Orthogonal Decomposition (POD) for model reduction of time dependent partial differential equations. The presented numerical results show the stability of proposed DR-RNN as an explicit reduced order technique. We also show significant gains in accuracy by increasing the depth of proposed DR-RNN similar to other applications of deep learning.

Motivation & Objective

• Motivate model reduction for nonlinear dynamical systems to enable fast design, optimization, and uncertainty quantification.
• Develop a physics-aware DR-RNN architecture inspired by iterative line search to minimize residuals of discretized ODEs/PDEs.
• Analyze temporal model reduction capability and compare to standard RNN/LSTM architectures.
• Demonstrate integration of DR-RNN with POD/DEIM for explicit reduced order modeling of time-dependent PDEs.

Proposed method

• Formulate DR-RNN as a stacked residual network that iteratively minimizes the residual from implicit time stepping of discretized dynamics (Eq. 19).
• Define layer updates y^(k+1) = y^(k) - w ∘ φ_h(U r^(k+1)) for k=1, and y^(k+1) = y^(k) - (η_k / sqrt(G_k+ε)) r^(k+1) for k>1, with residual r^(k+1) computed from the current guess (Eq. 20).
• Use tanh as activation and rmsprop-like updates for stability (Eq. 21–23).
• Set U as identity and train w and η_k while keeping W as identity; output is y_t+1^(RNN) = W^T y_t+1^K (Eq. 22).
• Combine DR-RNN with POD-Galerkin and POD-DEIM to obtain explicit reduced-order models with reduced complexity (discussion in Sec. 2–4).
• Train and validate DR-RNN against standard RNN/LSTM baselines on temporal model reduction tasks and compare accuracy and parameter count (Sec. 5.1).

Experimental results

Research questions

• RQ1 Can DR-RNN accurately emulate full-order nonlinear dynamical systems with significantly fewer parameters than standard RNNs?
• RQ2 Does stacking residual layers improve stability and accuracy for explicit reduced-order models with large time steps?
• RQ3 How does DR-RNN perform as a temporal model reducer compared to RNN and LSTM architectures of similar state dimension?
• RQ4 Can DR-RNN be effectively combined with POD/DEIM to form efficient projection-based reduced-order models for time-dependent PDEs?
• RQ5 What is the impact of increasing DR-RNN depth on accuracy and stability across test problems?

Key findings

• DR-RNN achieves considerably lower parameter counts than standard RNN/LSTM while matching or surpassing accuracy in the tested temporal reduction tasks.
• DR-RNN with residual layers (K=2 or 4) provides improved fit over RNN/LSTM in Problem 1, with substantially smaller MSE on training and testing data.
• DR-RNN can operate with larger time steps and still maintain accuracy, exhibiting stability as an explicit reduced-order technique.
• When incorporated with POD/DEIM, DR-RNN reduces computational effort from O(n^3) to O(n^2) per time step for nonlinear ODEs, and further to O(r^2) for ROM with reduced basis of size r.
• Increasing DR-RNN depth yields gains in accuracy, with deeper variants (e.g., DR-RNN_4) performing best on sample problems.
• A table of comparative results (Table 1) shows that DR-RNN variants achieve lower mse and dramatically fewer parameters than traditional RNN/LSTM baselines in the reported Problem 1.