[Paper Review] Convolutional Neural Operators for robust and accurate learning of PDEs

Bogdan Raonić, R. Molinaro+6 more|arXiv (Cornell University)|Feb 2, 2023

Neural Networks and Applications37 citations

TL;DR

The paper introduces Convolutional Neural Operators (CNOs), a CNN-based operator learning framework that preserves continuous-discrete equivalence to learn PDE solution operators with universality and strong empirical performance on diverse multi-scale PDE benchmarks.

ABSTRACT

Although very successfully used in conventional machine learning, convolution based neural network architectures -- believed to be inconsistent in function space -- have been largely ignored in the context of learning solution operators of PDEs. Here, we present novel adaptations for convolutional neural networks to demonstrate that they are indeed able to process functions as inputs and outputs. The resulting architecture, termed as convolutional neural operators (CNOs), is designed specifically to preserve its underlying continuous nature, even when implemented in a discretized form on a computer. We prove a universality theorem to show that CNOs can approximate operators arising in PDEs to desired accuracy. CNOs are tested on a novel suite of benchmarks, encompassing a diverse set of PDEs with possibly multi-scale solutions and are observed to significantly outperform baselines, paving the way for an alternative framework for robust and accurate operator learning. Our code is publicly available at https://github.com/bogdanraonic3/ConvolutionalNeuralOperator

Motivation & Objective

• Motivate learning PDE solution operators with data-driven surrogates to reduce computational cost for queries, uncertainty quantification, and optimization.
• Propose a structure-preserving CNN architecture (CNO) that maps bandlimited functions to bandlimited functions to avoid aliasing and preserve continuity.
• Prove a universal approximation theorem showing CNO can approximate PDE operators to arbitrary accuracy under suitable conditions.
• Evaluate CNO on a diverse Representative PDE Benchmarks (RPB) including linear, nonlinear, elliptic, hyperbolic, and parabolic PDEs with multi-scale solutions; compare against baselines and assess out-of-distribution generalization.

Proposed method

• Define operator learning on bandlimited function spaces to enable continuous-discrete equivalence (CDE).
• Instantiate CNO as a modified Operator U-Net with lifting P, convolution K_l, activation Sigma, and projection Q between bandlimited spaces.
• Ensure activation, up/downsampling, and ResNet-like blocks respect bandlimits to avoid aliasing and preserve multiscale content.
• Prove CNO is a Representation Equivalent Neural Operator (ReNO), ensuring resolution-invariant behavior and aliasing-free performance.
• Prove a universal approximation theorem: for any suitable PDE operator G†, there exists a CNO G approximating G† to any ε on bounded inputs.

Experimental results

Research questions

• RQ1 Can CNO approximate the solution operator of broad PDE classes to arbitrary accuracy within bandlimited spaces?
• RQ2 Does CNO satisfy representation equivalence and resolution-invariance to avoid aliasing across discretizations?
• RQ3 How does CNO compare to state-of-the-art operator learners (FNO, DeepONet, GT) on multi-scale PDE benchmarks?
• RQ4 Do CNOs generalize to out-of-distribution inputs and maintain performance across resolutions?

Key findings

• CNO consistently outperforms baselines across the Representative PDE Benchmarks (RPB) in both in-distribution and out-of-distribution tests except Allen-Cahn where FNO dominates.
• CNO achieves resolution-invariant behavior, unlike FNO and U-Net, demonstrating continuous-discrete equivalence.
• CNO exhibits favorable spectra with less aliasing and higher data efficiency, requiring fewer samples to reach a target error and training faster for comparable performance.
• CNO shows strong multi-scale handling and competitive or superior performance to baselines on Poisson, Wave, Transport, Navier-Stokes, Darcy, and Euler-related tasks.
• FNO, while strong on several tasks, can exhibit aliasing and non-invariance under resampling, highlighting the advantage of the CDE-preserving design.
• The study provides public code and a new suite of Representative PDE Benchmarks to benchmark operator learning methods.