[Paper Review] Challenges of learning multi-scale dynamics with AI weather models: Implications for stability and one solution

Ashesh Chattopadhyay, Sun, Y. Qiang+1 more|arXiv (Cornell University)|Apr 14, 2023

Meteorological Phenomena and Simulations14 citations

TL;DR

The paper identifies spectral bias as the universal cause of long-term instabilities in data-driven digital twins of turbulent flows and introduces FouRKS to achieve long-term, physically consistent stability in climate emulations. It demonstrates hundreds to tens of thousands of days of stable prediction on QG and ERA5 data.

ABSTRACT

Long-term stability and physical consistency are critical properties for AI-based weather models if they are going to be used for subseasonal-to-seasonal forecasts or beyond, e.g., climate change projection. However, current AI-based weather models can only provide short-term forecasts accurately since they become unstable or physically inconsistent when time-integrated beyond a few weeks or a few months. Either they exhibit numerical blow-up or hallucinate unrealistic dynamics of the atmospheric variables, akin to the current class of autoregressive large language models. The cause of the instabilities is unknown, and the methods that are used to improve their stability horizons are ad-hoc and lack rigorous theory. In this paper, we reveal that the universal causal mechanism for these instabilities in any turbulent flow is due to extit{spectral bias} wherein, extit{any} deep learning architecture is biased to learn only the large-scale dynamics and ignores the small scales completely. We further elucidate how turbulence physics and the absence of convergence in deep learning-based time-integrators amplify this bias, leading to unstable error propagation. Finally, using the quasi-geostrophic flow and European Center for Medium-Range Weather Forecasting (ECMWF) Reanalysis data as test cases, we bridge the gap between deep learning theory and numerical analysis to propose one mitigative solution to such unphysical behavior. We develop long-term physically-consistent data-driven models for the climate system and demonstrate accurate short-term forecasts, and hundreds of years of time-integration with accurate mean and variability.

Motivation & Objective

• Identify the universal cause of instabilities in data-driven digital twins of turbulent flows.
• Propose a physics-inspired, architecture-agnostic mitigation framework (FouRKS) for long-term stability.
• Demonstrate long-term stable and physically consistent emulations on QG and ERA5 datasets.
• Assess how FouRKS preserves mean, PDFs, and variability over extended forecasts.

Proposed method

• Analyze spectral bias as the source of instability in deep learning-based digital twins.
• Develop Fourier-based spectral regularization to penalize high-wavenumber errors during training.
• Incorporate a convergent 4th-order Runge-Kutta time integrator as a differentiable layer inside the model.
• Implement a self-supervised spectrum correction strategy during autoregressive prediction.
• Make FouRKS architecture-agnostic to work with any neural dynamical emulator predicting PDE residuals.

Figure 1: Long-term instabilities in a FourCastNet pathak2022fourcastnet and U-NET-based digital twin (section 4.3 ) trained on both $0.25^{\circ}$ and $2^{\circ}$ ERA5 data (section 4.1 ) and U-NET on QG simulations (section 4.2 ). (a) Z500 values from ERA5 on day 1 from year $2018$ . (b) correspon

Experimental results

Research questions

• RQ1 What is the fundamental cause of long-term instabilities in data-driven digital twins of turbulent flows?
• RQ2 Can a principled framework mitigate spectral bias and yield convergent, long-term predictions?
• RQ3 How does FouRKS perform in terms of physically meaningful long-term statistics (mean, PDF, variability) for QG and ERA5 data?
• RQ4 To what extent can FouRKS deliver stable predictions for hundreds to thousands of days?

Key findings

• Spectral bias is identified as the universal cause of instability in data-driven digital twins of turbulent flows.
• A Fourier-based spectral regularizer, RK4 integrator, and self-supervised spectrum correction synergistically enable long-term stability.
• FouRKS with a U-NET on the two-layer QG system achieves stable autoregressive emulation for 20,000 days.
• FouRKS with ERA5 data yields stable and physically consistent autoregressive predictions for up to 5,200 days.
• The framework produces predictions whose Fourier spectrum matches the true spectrum across scales, and preserves mean and PDFs of key variables.
• Compared to baseline models, FouRKS markedly improves long-term stability and physical realism of forecasts.

Figure 2: Schematics for each component of the FouRKS framework and the baseline U-NET. More details about each of the components can be found in section 4.3 and section 4.4 .