[Paper Review] Diffusion model based data generation for partial differential equations

Rucha Apte, Sheel Nidhan+2 more|arXiv (Cornell University)|Jun 19, 2023

Computational Physics and Python Applications4 citations

TL;DR

This paper proposes a diffusion model-based data generation framework for physics-based simulations, using denoising diffusion implicit models (DDIMs) to synthesize high-fidelity solutions to the 2D Poisson and Navier-Stokes equations. By applying physics-based constraints—PDE residual minimization and solver comparison—it achieves physically consistent, diverse, and statistically accurate synthetic data, with MSE values as low as 8×10⁻³ for Navier-Stokes and 1.5×10⁻⁴ for Poisson equations.

ABSTRACT

In a preliminary attempt to address the problem of data scarcity in physics-based machine learning, we introduce a novel methodology for data generation in physics-based simulations. Our motivation is to overcome the limitations posed by the limited availability of numerical data. To achieve this, we leverage a diffusion model that allows us to generate synthetic data samples and test them for two canonical cases: (a) the steady 2-D Poisson equation, and (b) the forced unsteady 2-D Navier-Stokes (NS) {vorticity-transport} equation in a confined box. By comparing the generated data samples against outputs from classical solvers, we assess their accuracy and examine their adherence to the underlying physics laws. In this way, we emphasize the importance of not only satisfying visual and statistical comparisons with solver data but also ensuring the generated data's conformity to physics laws, thus enabling their effective utilization in downstream tasks.

Motivation & Objective

• Address data scarcity in physics-based machine learning by generating synthetic simulation data using diffusion models.
• Ensure generated data adheres to underlying physical laws, not just statistical or visual similarity to solver outputs.
• Demonstrate the feasibility of using unconditional diffusion models for generating physically consistent solutions to canonical PDEs.
• Evaluate the effectiveness of physics-based filtering to select high-quality, physically valid samples from generated distributions.
• Lay the foundation for scalable, generalizable data generation in scientific machine learning, with future extensions to super-resolution and parameter conditioning.

Proposed method

• Utilizes a denoising diffusion implicit model (DDIM) with a U-Net architecture to generate synthetic PDE solutions from Gaussian noise.
• Employs a cosine noise schedule over 1000 forward steps and 500 reverse sampling steps to accelerate generation while preserving quality.
• For the 2D Poisson equation, both the solution $ u $ and source term $ f $ are input as separate channels to the U-Net.
• For time-dependent Navier-Stokes problems, multiple timesteps are encoded as separate channels in the U-Net input.
• Applies physics-based filtering: for Poisson, selects samples with minimal PDE residual MSE ($ |\nabla^2 u - f| $); for Navier-Stokes, compares generated snapshots to classical solver outputs.
• Uses a 500-step DDIM sampling process to generate denoised outputs from random noise, enabling efficient and diverse sample generation.

Figure 1: Four $[f_{\mathrm{generated}},u_{\mathrm{generated}}]$ pairs generated from diffusion model (first two columns from left), $u_{\mathrm{solver}}$ generated from the corresponding $f$ (third column), PDE residual calculated using generated $[f,u]$ pairs (fourth column). $\mathrm{MSE}$ of PDE

Experimental results

Research questions

• RQ1 Can diffusion models generate synthetic data for PDEs that are both visually and statistically similar to solutions from classical solvers?
• RQ2 To what extent can diffusion-generated data satisfy the underlying physical laws without explicit physics encoding during training?
• RQ3 How effective are physics-based post-processing filters (e.g., PDE residual or solver comparison) in selecting physically consistent samples?
• RQ4 Can the generated data exhibit sufficient diversity and novelty to be useful for downstream ML tasks in scientific computing?
• RQ5 What are the quantitative and qualitative differences between diffusion-generated and solver-generated solutions for canonical PDEs like Poisson and Navier-Stokes?

Key findings

• The diffusion model successfully generated visually and statistically consistent samples for both the 2D Poisson and 2D Navier-Stokes equations.
• For the 2D Poisson equation, the PDE residual MSE across the grid was as low as 1.5×10⁻⁴, indicating strong adherence to the governing equation.
• For the 2D Navier-Stokes vorticity-transport equation, the MSE between diffusion-generated and solver-generated snapshots was approximately 8×10⁻³ for the first four time steps.
• In cases with higher MSE (0.045), the flow dynamics remained qualitatively consistent, with vortex roll-up around [0.5, 0.5] observed in both generated and solver outputs.
• The mean vorticity field from diffusion-generated data qualitatively reproduced the inclined patch structure induced by the forcing function, though positive vorticity patches were more dominant than in solver data.
• Visual inspection confirmed that the generated samples exhibited high diversity and distinct features, indicating effective exploration of the solution space.

Figure 2: Diffusion- and solver-generated vorticity snapshots for 2-D forced NS system. Top two rows correspond to $\mathrm{MSE}$ $\approx 8\times 10^{-3}$ between solver-generated and diffusion-generated snapshots. Bottom two rows correspond to $\mathrm{MSE}\approx 0.045$ .