[Paper Review] Derivative-Informed Projected Neural Networks for High-Dimensional Parametric Maps Governed by PDEs

Thomas O’Leary-Roseberry, Umberto Villa+2 more|arXiv (Cornell University)|Nov 30, 2020

Model Reduction and Neural Networks57 references22 citations

TL;DR

This paper proposes Derivative-Informed Projected Neural Networks (DIPNets) to efficiently surrogate high-dimensional parametric maps governed by PDEs. By leveraging Jacobian information to construct a derivative-informed active subspace for inputs and proper orthogonal decomposition for outputs, DIPNets achieve high accuracy with minimal training data, demonstrating superior generalization and dimension-independent performance in challenging many-query PDE problems.

ABSTRACT

Many-query problems, arising from uncertainty quantification, Bayesian inversion, Bayesian optimal experimental design, and optimization under uncertainty-require numerous evaluations of a parameter-to-output map. These evaluations become prohibitive if this parametric map is high-dimensional and involves expensive solution of partial differential equations (PDEs). To tackle this challenge, we propose to construct surrogates for high-dimensional PDE-governed parametric maps in the form of projected neural networks that parsimoniously capture the geometry and intrinsic low-dimensionality of these maps. Specifically, we compute Jacobians of these PDE-based maps, and project the high-dimensional parameters onto a low-dimensional derivative-informed active subspace; we also project the possibly high-dimensional outputs onto their principal subspace. This exploits the fact that many high-dimensional PDE-governed parametric maps can be well-approximated in low-dimensional parameter and output subspace. We use the projection basis vectors in the active subspace as well as the principal output subspace to construct the weights for the first and last layers of the neural network, respectively. This frees us to train the weights in only the low-dimensional layers of the neural network. The architecture of the resulting neural network captures to first order, the low-dimensional structure and geometry of the parametric map. We demonstrate that the proposed projected neural network achieves greater generalization accuracy than a full neural network, especially in the limited training data regime afforded by expensive PDE-based parametric maps. Moreover, we show that the number of degrees of freedom of the inner layers of the projected network is independent of the parameter and output dimensions, and high accuracy can be achieved with weight dimension independent of the discretization dimension.

Motivation & Objective

• Address the challenge of high computational cost in many-query problems involving high-dimensional parametric PDEs.
• Develop surrogate models that are accurate, efficient, and scalable despite limited training data due to expensive PDE solves.
• Exploit intrinsic low-dimensional structure in both input parameters and output maps to reduce model complexity.
• Construct neural network architectures that are dimension-independent and generalize well in low-data regimes.
• Provide a framework for transfer learning and multi-fidelity modeling in PDE-constrained inference and optimization.

Proposed method

• Compute the Jacobian of the PDE-governed parametric map to identify derivative-informed active subspaces for input parameter reduction.
• Use proper orthogonal decomposition (POD) on output samples to identify a low-dimensional principal output subspace.
• Project high-dimensional inputs and outputs onto their respective low-dimensional subspaces using basis vectors from active subspace and POD.
• Construct a neural network where the first and last layer weights are fixed using the input and output projection bases, reducing trainable parameters to only the hidden layers.
• Train only the low-dimensional hidden layers, achieving a network architecture that captures the first-order geometry of the parametric map.
• Ensure infinite-dimensional consistency by using continuous PDE-based sensitivity information, avoiding discretization artifacts.

Experimental results

Research questions

• RQ1 Can derivative-informed dimension reduction improve surrogate model accuracy in high-dimensional PDE-governed parametric maps with limited training data?
• RQ2 How does using Jacobian-based active subspaces compare to Karhunen-Loève-based or random projections in input dimension reduction for neural network surrogates?
• RQ3 To what extent does the proposed DIPNet architecture maintain accuracy across different PDE problems with varying solution characteristics?
• RQ4 Can the number of trainable parameters in the neural network be kept independent of the discretization dimension while preserving high accuracy?
• RQ5 Does the use of derivative information in the input projection lead to better generalization than data-driven or random projections?

Key findings

• DIPNets significantly outperform full-space dense neural networks in the low-data regime, demonstrating superior generalization capability.
• The DIPNet strategy achieved high accuracy on both convection-diffusion-reaction and Helmholtz problems, outperforming KLE-based and random projector alternatives.
• In the Helmholtz problem, where KLE failed to reduce projection error effectively, DIPNets maintained strong performance due to the ability of active subspaces to resolve localized, high-frequency features.
• The number of degrees of freedom in the inner layers of the DIPNet is independent of the input and output dimensions, enabling dimension-independent complexity.
• Projection error analysis confirms that the active subspace and POD bases are effective for reducing the parametric map's intrinsic dimensionality with minimal loss of information.
• The framework supports extensions to transfer learning and multi-fidelity modeling due to its infinite-dimensional consistency and structure-aware design.