[论文解读] Toward a Better Understanding of Fourier Neural Operators from a Spectral Perspective

Shaoxiang Qin, Fuyuan Lyu+8 more|arXiv (Cornell University)|Apr 10, 2024

Neural Networks and Applications被引用 10

一句话总结

本论文从谱角度分析 Fourier Neural Operators (FNOs)，表明 FNOs 能较好学习低频内容，但在高频细节上有所缺失，并提出 SpecBoost—a sequential two-FNO ensemble—to 更好地捕捉高频信息，从而提高 PDE 预测的准确性。

ABSTRACT

In solving partial differential equations (PDEs), Fourier Neural Operators (FNOs) have exhibited notable effectiveness. However, FNO is observed to be ineffective with large Fourier kernels that parameterize more frequencies. Current solutions rely on setting small kernels, restricting FNO's ability to capture complex PDE data in real-world applications. This paper offers empirical insights into FNO's difficulty with large kernels through spectral analysis: FNO exhibits a unique Fourier parameterization bias, excelling at learning dominant frequencies in target data while struggling with non-dominant frequencies. To mitigate such a bias, we propose SpecB-FNO to enhance the capture of non-dominant frequencies by adopting additional residual modules to learn from the previous ones' prediction residuals iteratively. By effectively utilizing large Fourier kernels, SpecB-FNO achieves better prediction accuracy on diverse PDE applications, with an average improvement of 50%.

研究动机与目标

• Explain why FNOs outperform CNNs via spectral analysis, focusing on low-frequency learning ability.
• Identify the high-frequency learning limitations of FNOs caused by global Fourier filters and mode truncation.
• Propose and validate SpecBoost, an ensemble framework to recover high-frequency information from FNO residuals.
• Demonstrate SpecBoost's effectiveness across multiple PDE tasks and data formats while highlighting memory efficiency.

提出的方法

• Perform spectral analysis of prediction errors to compare FNO and CNN performance on Navier-Stokes data."
• Show that FNO’s low-frequency bias arises from global Fourier filters and frequency truncation.
• Introduce SpecBoost: train a second FNO on the first FNO’s residual to capture high-frequency information."
• Use sequential training where the first FNO predicts y, the second FNO learns the residual r = y - ŷ, and final output is ŷ + ṙ.
• Demonstrate SpecBoost compatibility with various FNO variants and data formats (regular grids, irregular grids, spherical, clouds, general geometries).
• Evaluate memory efficiency of SpecBoost versus solo models while analyzing spectral behavior.

实验结果

研究问题

• RQ1 Why does FNO outperform CNNs in PDE tasks from a spectral perspective?
• RQ2 What causes FNO’s difficulty in learning high-frequency information, and how can this be mitigated?
• RQ3 Can a secondary model trained on the first model’s residual improve high-frequency capture without excessive cost?
• RQ4 How does SpecBoost perform across Navier-Stokes and Darcy flow PDEs, and how does data resolution affect performance?
• RQ5 Is SpecBoost compatible with different neural operator architectures and data formats?

主要发现

• FNO more effectively learns low-frequency information than CNNs, explaining its superiority for PDEs.
• FNO exhibits a low-frequency bias due to global Fourier filters and high-frequency mode truncation, limiting high-frequency learning.
• SpecBoost, a two-FNO ensemble trained sequentially, significantly reduces errors by learning residual high-frequency content (up to 71% improvement on Navier-Stokes for ν=1e-3).
• On Darcy flow, SpecBoost achieves up to 61% error reduction across resolutions S=85–421.
• SpecBoost improves PDE data reconstruction and compression tasks compared to solo FNO variants, with notable gains for FNO-SR and FNO-AE.
• A memory-efficient training advantage is observed: SpecBoost can reduce max memory usage by about 35% at similar total layer counts.