[Paper Review] Deep learning applied to computational mechanics: A comprehensive review, state of the art, and the classics

L. Vu‐Quoc, Alexander Humer|arXiv (Cornell University)|Dec 18, 2022

Computational Physics and Python Applications5 citations

TL;DR

This comprehensive review integrates deep learning (DL) with computational mechanics, covering hybrid and pure ML methods for solving PDEs, model-order reduction, and numerical integration. It presents state-of-the-art DL architectures—LSTM, transformers, PINNs—with detailed optimization techniques, kernel methods, and practical applications in solid and fluid mechanics, emphasizing reproducibility and critical analysis of AI limitations in science.

ABSTRACT

Three recent breakthroughs due to AI in arts and science serve as motivation: An award winning digital image, protein folding, fast matrix multiplication. Many recent developments in artificial neural networks, particularly deep learning (DL), applied and relevant to computational mechanics (solid, fluids, finite-element technology) are reviewed in detail. Both hybrid and pure machine learning (ML) methods are discussed. Hybrid methods combine traditional PDE discretizations with ML methods either (1) to help model complex nonlinear constitutive relations, (2) to nonlinearly reduce the model order for efficient simulation (turbulence), or (3) to accelerate the simulation by predicting certain components in the traditional integration methods. Here, methods (1) and (2) relied on Long-Short-Term Memory (LSTM) architecture, with method (3) relying on convolutional neural networks. Pure ML methods to solve (nonlinear) PDEs are represented by Physics-Informed Neural network (PINN) methods, which could be combined with attention mechanism to address discontinuous solutions. Both LSTM and attention architectures, together with modern and generalized classic optimizers to include stochasticity for DL networks, are extensively reviewed. Kernel machines, including Gaussian processes, are provided to sufficient depth for more advanced works such as shallow networks with infinite width. Not only addressing experts, readers are assumed familiar with computational mechanics, but not with DL, whose concepts and applications are built up from the basics, aiming at bringing first-time learners quickly to the forefront of research. History and limitations of AI are recounted and discussed, with particular attention at pointing out misstatements or misconceptions of the classics, even in well-known references. Positioning and pointing control of a large-deformable beam is given as an example.

Motivation & Objective

• To bridge deep learning with computational mechanics by reviewing hybrid and pure ML methods for solving PDEs and reducing simulation costs.
• To provide a unified, accessible review of DL fundamentals—from backpropagation to modern optimizers—targeting researchers in computational mechanics unfamiliar with DL.
• To critically examine misconceptions and limitations in AI, especially regarding reproducibility, generalization, and the overstatement of adaptive optimizers.
• To present practical applications, including LSTM-based constitutive modeling, autoencoder-based ROMs, and Physics-Informed Neural Networks (PINNs) with attention mechanisms.
• To advocate for methodological rigor in AI research by highlighting transparency issues and the need for verifiable, reproducible results in scientific DL applications.

Proposed method

• Uses Long Short-Term Memory (LSTM) networks to model rate-dependent constitutive behavior in porous media and as time-integrators in reduced-order models.
• Applies Physics-Informed Neural Networks (PINNs) with attention mechanisms to solve nonlinear PDEs, especially those with discontinuous solutions.
• Employs autoencoders and hyper-reduction techniques using gappy data to enable nonlinear manifold-based model-order reduction (MOR) in fluid and solid mechanics.
• Reviews and derives stochastic optimization methods, including SGD with momentum, learning rate decay, and adaptive methods (Adam, AdamW), linking them to simulated annealing and stochastic differential equations.
• Introduces kernel machines and Gaussian processes with infinite-width limits to support advanced non-parametric modeling and uncertainty quantification.
• Provides detailed pseudocodes and derivations for optimization algorithms, including step-length decay and batch-size scaling as alternatives to learning rate decay.

Experimental results

Research questions

• RQ1 How can deep learning be effectively combined with traditional PDE solvers to reduce computational cost in computational mechanics?
• RQ2 What are the strengths and limitations of adaptive optimizers like Adam compared to carefully tuned SGD in deep learning for physics-based modeling?
• RQ3 How do attention mechanisms and transformer architectures improve the solution of PDEs with discontinuities or sharp gradients?
• RQ4 To what extent can autoencoders and reduced-order models trained with DL replace high-fidelity simulations in fluid and solid mechanics?
• RQ5 Why is reproducibility in AI research—especially in scientific applications—undermined by poor documentation of hyperparameters and training protocols?

Key findings

• LSTM networks effectively model complex, rate-dependent constitutive behavior in multi-scale geomechanics and fluid-filled porous media, enabling efficient reduced-order simulations.
• PINNs with attention mechanisms can handle discontinuous solutions in PDEs, improving accuracy over standard PINNs in challenging fluid dynamics and solid mechanics problems.
• Adaptive optimizers like Adam show marginal generalization benefits over well-tuned SGD with learning rate decay, challenging their widespread use in scientific DL.
• Step-length decay in stochastic optimization is mathematically equivalent to simulated annealing via stochastic differential equations, suggesting that increasing batch size can replace learning rate decay for better convergence.
• Reproducibility in AI is severely compromised by missing hyperparameter details—e.g., learning rate, batch size, and optimizer settings—highlighted by a case study on breast cancer detection with insufficient documentation.
• Kernel machines, including Gaussian processes with infinite-width limits, provide robust uncertainty estimates and form a foundation for modern deep learning theory, especially in non-parametric modeling.