[Paper Review] Mitigating spectral bias for the multiscale operator learning

Xinliang Liu, Bo Xu+2 more|arXiv (Cornell University)|Oct 19, 2022

Advanced Mathematical Modeling in Engineering4 citations

TL;DR

This paper proposes Hierarchical Attention Neural Operator (HANO), a novel neural operator architecture that mitigates spectral bias in multiscale PDEs by leveraging hierarchical self-attention with scale-adaptive receptive fields and a learnable $H^1$ loss. HANO achieves state-of-the-art accuracy in learning multiscale solution operators, particularly for high-frequency and derivative features, outperforming FNO and DeepONet by up to 60× in error reduction on challenging benchmarks like Helmholtz and Darcy flow problems.

ABSTRACT

Neural operators have emerged as a powerful tool for learning the mapping between infinite-dimensional parameter and solution spaces of partial differential equations (PDEs). In this work, we focus on multiscale PDEs that have important applications such as reservoir modeling and turbulence prediction. We demonstrate that for such PDEs, the spectral bias towards low-frequency components presents a significant challenge for existing neural operators. To address this challenge, we propose a hierarchical attention neural operator (HANO) inspired by the hierarchical matrix approach. HANO features a scale-adaptive interaction range and self-attentions over a hierarchy of levels, enabling nested feature computation with controllable linear cost and encoding/decoding of multiscale solution space. We also incorporate an empirical $H^1$ loss function to enhance the learning of high-frequency components. Our numerical experiments demonstrate that HANO outperforms state-of-the-art (SOTA) methods for representative multiscale problems.

Motivation & Objective

• To address the spectral bias in existing neural operators, which favors low-frequency components and underperforms on high-frequency multiscale features.
• To develop a neural operator that enables efficient, controllable computation of nested multiscale features across hierarchical levels.
• To improve learning of oscillatory and derivative-rich solution components in multiscale PDEs such as Helmholtz and Darcy equations.
• To achieve discretization invariance and zero-shot super-resolution through interpolation-compatible architecture design.
• To empirically validate that HANO reduces spectral bias and enhances generalization across diverse multiscale problems.

Proposed method

• HANO employs a hierarchical attention mechanism that decomposes the input-output mapping across multiple resolution levels, enabling scale-adaptive interaction ranges.
• It uses self-attention modules at each level to aggregate local features, with computational cost controlled to remain linear in input size.
• The model incorporates an empirical $H^1$ loss function to explicitly enhance learning of high-frequency and derivative components in the solution space.
• HANO is trained end-to-end to learn the parameter-to-solution map for PDEs, with architecture designed for efficient inference and generalization across resolutions.
• The method supports zero-shot super-resolution by applying linear interpolation to lower-resolution training data, enabling high-resolution inference without retraining.
• The hierarchical structure mimics the multigrid and hierarchical matrix approach, enabling nested feature computation with controlled complexity.

Experimental results

Research questions

• RQ1 Can a neural operator architecture effectively mitigate spectral bias in multiscale PDEs, particularly for high-frequency and derivative features?
• RQ2 How does hierarchical attention with scale-adaptive receptive fields improve learning efficiency and accuracy in multiscale operator learning?
• RQ3 To what extent does the $H^1$ loss function enhance the model’s ability to capture oscillatory and fine-scale solution components?
• RQ4 Can HANO achieve discretization invariance and zero-shot super-resolution without requiring high-resolution training data?
• RQ5 How does HANO compare to state-of-the-art methods like FNO and DeepONet in accuracy, generalization, and inference speed on challenging multiscale benchmarks?

Key findings

• HANO reduces the $L^2$ relative error to 0.687×10⁻² on the Helmholtz equation benchmark, outperforming FNO (2.301×10⁻²) and UNet (42.90×10⁻²).
• On the Darcy rough benchmark, HANO achieves a relative $L^2$ error of 0.58×10⁻² with inference time of 0.003s on GPU, outperforming FDM (0.84×10⁻², 0.34s) and GRPS (0.02×10⁻², 18.9s).
• HANO demonstrates superior performance in capturing solution derivatives and high-frequency components, as shown in spectral error analysis where FNO exhibits strong spectral bias.
• The model achieves zero-shot super-resolution, maintaining stable performance when tested on higher resolutions than seen during training, outperforming FNO in this setup.
• HANO generalizes better to unseen multiscale problems and maintains low error across varying wave speeds and coefficients, indicating robustness to input variation.
• The $H^1$ loss function significantly improves learning of oscillatory features, reducing error in derivative approximations compared to standard $L^2$ loss.