[Paper Review] PDE-Net: Learning PDEs from Data

Zichao Long, Yiping Lu+2 more|arXiv (Cornell University)|Oct 26, 2017

Model Reduction and Neural Networks5 references219 citations

TL;DR

PDE-Net learns differential operators via constrained learnable filters and a neural network to predict dynamics and uncover underlying PDE models from data.

ABSTRACT

In this paper, we present an initial attempt to learn evolution PDEs from data. Inspired by the latest development of neural network designs in deep learning, we propose a new feed-forward deep network, called PDE-Net, to fulfill two objectives at the same time: to accurately predict dynamics of complex systems and to uncover the underlying hidden PDE models. The basic idea of the proposed PDE-Net is to learn differential operators by learning convolution kernels (filters), and apply neural networks or other machine learning methods to approximate the unknown nonlinear responses. Comparing with existing approaches, which either assume the form of the nonlinear response is known or fix certain finite difference approximations of differential operators, our approach has the most flexibility by learning both differential operators and the nonlinear responses. A special feature of the proposed PDE-Net is that all filters are properly constrained, which enables us to easily identify the governing PDE models while still maintaining the expressive and predictive power of the network. These constrains are carefully designed by fully exploiting the relation between the orders of differential operators and the orders of sum rules of filters (an important concept originated from wavelet theory). We also discuss relations of the PDE-Net with some existing networks in computer vision such as Network-In-Network (NIN) and Residual Neural Network (ResNet). Numerical experiments show that the PDE-Net has the potential to uncover the hidden PDE of the observed dynamics, and predict the dynamical behavior for a relatively long time, even in a noisy environment.

Motivation & Objective

• Motivate data-driven discovery of governing PDEs for complex systems with limited prior knowledge.
• Develop a deep feed-forward network that can both predict dynamics and reveal hidden PDE forms.
• Learn differential operators via trainable, constrained convolution kernels alongside a nonlinear response function F.

Proposed method

• Represent PDEs in a discrete, learnable form where u_t = F(x,y,u, u_x, u_y, u_xx, u_xy, u_yy, ...) .
• Use convolution kernels to learn discretizations of differential operators; learn F with a point-wise neural network.
• Constrain filters through moment matrices tied to sum rules to link learned operators to differential operators.
• Stack multiple delta-t blocks to enforce longer-time stability and enable long-term predictions.
• Share parameters across layers to reduce memory and enforce consistency.
• Initialize filters to correspond to known differential operators and gradually release constraints during training.

Experimental results

Research questions

• RQ1 Can PDE-Net accurately predict complex dynamics over long times from data with noise?
• RQ2 Can PDE-Net uncover the underlying PDE structure (coefficients and operators) from observed dynamics?
• RQ3 How do filter size and network depth affect predictive stability and PDE identification?
• RQ4 Does constraining filters via moment matrices aid identifiability of the governing PDEs?

Key findings

• PDE-Net achieves long-time prediction, outperforming Frozen-PDE-Net, especially with larger 7x7 filters.
• Increasing the number of delta-t blocks improves stability and accuracy of long-term predictions.
• Learned coefficients approximately match the true PDE coefficients for linear problems, with some oscillations due to noise.
• Larger filters (7x7) extend reliable prediction horizon compared to smaller filters (5x5).
• PDE-Net can uncover hidden PDEs in linear tests by identifying near-zero coefficients for non-present terms.
• Layer-wise training with shared parameters and filter constraints enhances learning efficiency and identifiability.