[Paper Review] Physics-informed Neural Networks with Unknown Measurement Noise

Philipp Pilar, Niklas Wahlström|arXiv (Cornell University)|Nov 28, 2022

Model Reduction and Neural Networks4 citations

TL;DR

This paper proposes PINN-EBM, a physics-informed neural network framework that jointly trains an energy-based model (EBM) to learn unknown, non-Gaussian measurement noise distributions, enabling robust PDE solution and parameter identification under non-ideal data conditions. The method significantly outperforms standard PINNs in accuracy and noise distribution recovery when noise is non-Gaussian or has non-zero mean.

ABSTRACT

Physics-informed neural networks (PINNs) constitute a flexible approach to both finding solutions and identifying parameters of partial differential equations. Most works on the topic assume noiseless data, or data contaminated with weak Gaussian noise. We show that the standard PINN framework breaks down in case of non-Gaussian noise. We give a way of resolving this fundamental issue and we propose to jointly train an energy-based model (EBM) to learn the correct noise distribution. We illustrate the improved performance of our approach using multiple examples.

Motivation & Objective

• Address the failure of standard physics-informed neural networks (PINNs) when trained on data with non-Gaussian or non-zero mean measurement noise.
• Overcome the limitations of least-squares loss in PINNs under non-Gaussian noise, which leads to biased parameter estimates and poor generalization.
• Develop a joint training framework that simultaneously learns the PDE solution, parameters, and the true noise probability density function (PDF) from noisy measurements.
• Enable system identification in real-world scenarios where measurement noise is complex, asymmetric, or contains systematic biases.
• Provide a method that not only improves predictive performance but also reveals previously undetected systematic errors in data collection processes.

Proposed method

• Integrate an energy-based model (EBM) into the PINN framework to model the unknown measurement noise distribution directly from residuals between predictions and noisy observations.
• Use the EBM to estimate the likelihood of measurement residuals, which replaces the standard least-squares data loss in PINNs with a more robust likelihood-based loss.
• Train the PINN and EBM jointly via backpropagation on a total loss combining PDE residual loss and EBM-based data likelihood loss.
• Formulate the total loss as a weighted sum of the physics-informed PDE loss and the negative log-likelihood (NLL) of the measurements under the EBM-learned noise distribution.
• Allow the EBM to capture complex noise characteristics such as heavy tails, asymmetry, or multimodality, including cases with multiple sensor biases.
• Use validation data to tune the weighting hyperparameter (ω) between PDE and data losses, ensuring balanced optimization.

Experimental results

Research questions

• RQ1 Can standard PINNs reliably identify PDE parameters when measurements are corrupted by non-Gaussian noise?
• RQ2 How does the performance of PINNs degrade under non-Gaussian noise compared to Gaussian noise, and what causes this degradation?
• RQ3 Can an energy-based model (EBM) effectively learn the true noise distribution from residuals in a PINN setting, even when the noise is unknown and non-Gaussian?
• RQ4 Does jointly training a PINN with an EBM improve parameter estimation and solution accuracy compared to standard PINN or PINN with fixed noise assumptions?
• RQ5 Can the EBM component detect systematic biases or multiple noise sources in measurement data, such as those arising from merged sensor readings?

Key findings

• The standard PINN framework fails under non-Gaussian noise, particularly when noise has non-zero mean or heavy tails, due to the use of least-squares loss.
• PINN-EBM significantly outperforms standard PINN and PINN-off in parameter recovery for the Navier-Stokes equations, reducing the error in λ₁ by a factor of ~20 (from 23.10 to 1.19 in 100|Δλ₁|).
• For the Navier-Stokes case with 3G noise, PINN-EBM achieves a 0.06 RMSE, compared to 0.20 for standard PINN, indicating superior solution accuracy.
• The NLL on validation data is -0.03 for PINN-EBM, compared to 0.40 for standard PINN, showing that the EBM learns a more accurate noise model.
• The method successfully learns the true multimodal noise distribution (e.g., Gaussian mixture with two peaks), including a bias component from a faulty sensor.
• The EBM component enables detection of previously unrecognized systematic errors in data, such as a second peak in the noise PDF indicating sensor bias.