[Paper Review] Physics-informed Neural Networks with Periodic Activation Functions for Solute Transport in Heterogeneous Porous Media

Salah A Faroughi, Soltanmohammad, Ramin+3 more|arXiv (Cornell University)|Dec 17, 2022

Model Reduction and Neural Networks4 citations

TL;DR

This study proposes physics-informed neural networks (PiNNs) with periodic activation functions (sin) to accelerate solute transport simulations in heterogeneous porous media governed by the advection-dispersion equation. By embedding physical laws directly into the loss function and using sin activations, the method achieves up to two orders of magnitude higher accuracy and two times faster training than tanh-based PiNNs, while reducing inference time by three orders of magnitude compared to finite element methods.

ABSTRACT

Simulating solute transport in heterogeneous porous media poses computational challenges due to the high-resolution meshing required for traditional solvers. To overcome these challenges, this study explores a mesh-free method based on deep learning to accelerate solute transport simulation. We employ Physics-informed Neural Networks (PiNN) with a periodic activation function to solve solute transport problems in both homogeneous and heterogeneous porous media governed by the advection-dispersion equation. Unlike traditional neural networks that rely on large training datasets, PiNNs use strong-form mathematical models to constrain the network in the training phase and simultaneously solve for multiple dependent or independent field variables, such as pressure and solute concentration fields. To demonstrate the effectiveness of using PiNNs with a periodic activation function to resolve solute transport in porous media, we construct PiNNs using two activation functions, sin and tanh, for seven case studies, including 1D and 2D scenarios. The accuracy of the PiNNs' predictions is then evaluated using absolute point error and mean square error metrics and compared to the ground truth solutions obtained analytically or numerically. Our results demonstrate that the PiNN with sin activation function, compared to tanh activation function, is up to two orders of magnitude more accurate and up to two times faster to train, especially in heterogeneous porous media. Moreover, PiNN's simultaneous predictions of pressure and concentration fields can reduce computational expenses in terms of inference time by three orders of magnitude compared to FEM simulations for two-dimensional cases.

Motivation & Objective

• To address the computational inefficiency of traditional solvers like FEM in simulating solute transport in heterogeneous porous media due to high-resolution meshing requirements.
• To develop a mesh-free deep learning approach that integrates physical laws directly into neural network training for accurate and efficient simulation.
• To evaluate the performance of PiNNs with periodic (sin) versus non-periodic (tanh) activation functions in predicting solute concentration and pressure fields.
• To demonstrate the capability of PiNNs to simultaneously predict multiple field variables (pressure and concentration) with high accuracy and low inference cost.
• To identify optimal hyperparameters and assess convergence stability in complex, heterogeneous domains using iterative random-search tuning.

Proposed method

• Employ physics-informed neural networks (PiNNs) that enforce the advection-dispersion equation, initial conditions, and boundary conditions as part of the loss function through automatic differentiation.
• Use periodic activation functions (sin) and compare them to the standard tanh activation function in PiNN architectures for solute transport modeling.
• Train the PiNNs using a loss function composed of residual terms from the PDE, initial conditions, and boundary conditions, with adaptive weighting to balance competing loss terms.
• Simultaneously predict both pressure and solute concentration fields within a single PiNN framework to reduce computational cost.
• Utilize iterative random-search hyperparameter tuning to optimize network architecture and hyperparameters for each case study.
• Validate predictions against analytical solutions or high-fidelity FEM simulations using absolute point error and mean square error metrics.

Experimental results

Research questions

• RQ1 How does the choice of activation function (sin vs. tanh) affect the accuracy and training speed of PiNNs in simulating solute transport in porous media?
• RQ2 Can PiNNs with periodic activation functions achieve higher accuracy and faster convergence than standard PiNNs in heterogeneous porous media with complex flow structures?
• RQ3 To what extent can PiNNs simultaneously predict pressure and concentration fields with reduced inference time compared to traditional FEM solvers?
• RQ4 What are the computational and convergence challenges in training PiNNs for multi-field problems in heterogeneous domains, and how can they be mitigated?
• RQ5 How does the inference speed-up of PiNNs scale with mesh resolution and heterogeneity levels in porous media?

Key findings

• The PiNN with sin activation function achieved up to two orders of magnitude higher accuracy than the tanh-based PiNN across all 1D and 2D case studies.
• The sin-based PiNN trained up to two times faster than the tanh-based counterpart, particularly in heterogeneous media with complex flow patterns.
• PiNN inference was up to 1,458.7 times faster than FEM in 2D homogeneous porous media, with speed-up factors of 1,399.6x to 1,402.3x in heterogeneous cases.
• Simultaneous prediction of pressure and concentration fields using PiNN reduced inference time by three orders of magnitude compared to FEM in 2D cases.
• Despite improved performance, PiNN training faced challenges including gradient vanishing and instability due to competing loss terms, requiring careful hyperparameter tuning.
• The method demonstrated robustness across varying degrees of permeability heterogeneity, maintaining high accuracy even in domains with distinct length scales and complex flow networks.