[Paper Review] Inductive biases in deep learning models for weather prediction

Jannik Thuemmel, Matthias Karlbauer+9 more|arXiv (Cornell University)|Apr 6, 2023

Hydrological Forecasting Using AI15 citations

TL;DR

The paper reviews inductive biases in six state-of-the-art DLWP models, analyzing data choices, objectives, losses, architectures, and optimisers, and discusses future directions with foundation models and physics-informed biases.

ABSTRACT

Deep learning has gained immense popularity in the Earth sciences as it enables us to formulate purely data-driven models of complex Earth system processes. Deep learning-based weather prediction (DLWP) models have made significant progress in the last few years, achieving forecast skills comparable to established numerical weather prediction models with comparatively lesser computational costs. In order to train accurate, reliable, and tractable DLWP models with several millions of parameters, the model design needs to incorporate suitable inductive biases that encode structural assumptions about the data and the modelled processes. When chosen appropriately, these biases enable faster learning and better generalisation to unseen data. Although inductive biases play a crucial role in successful DLWP models, they are often not stated explicitly and their contribution to model performance remains unclear. Here, we review and analyse the inductive biases of state-of-the-art DLWP models with respect to five key design elements: data selection, learning objective, loss function, architecture, and optimisation method. We identify the most important inductive biases and highlight potential avenues towards more efficient and probabilistic DLWP models.

Motivation & Objective

• Motivate the use of DL-based weather prediction (DLWP) as a data-driven alternative to traditional NWP under changing climate conditions.
• Systematically identify and articulate the inductive biases embedded in current top DLWP models.
• Assess how design choices across data, objectives, losses, architectures, and optimization affect learning, generalisation, and uncertainty.
• Outline future avenues, including foundation models and explicit physics-informed bias integration, to enhance DLWP performance.

Proposed method

• Survey six high-performing DLWP models (R21, E21, W21, P22, K22, H22) and map their design choices to inductive biases.
• Decompose each model along five design elements: data selection, learning objective, loss components, neural architecture, and optimization strategy.
• Diagram how each design choice encodes assumptions about atmospheric dynamics and subgrid processes.
• Compare how iterative vs direct forecasting, probabilistic vs deterministic outputs, and residual vs absolute prediction strategies influence learning biases.
• Discuss loss functions, normalization, and uncertainty modelling to align optimisation with verification metrics.

Experimental results

Research questions

• RQ1 What inductive biases are encoded by data selection, forecasting objectives, loss functions, architectures, and optimization in the six DLWP models?
• RQ2 How do these biases influence learning efficiency, generalisation, and uncertainty quantification across short- to mid-range forecasts?
• RQ3 What future directions (e.g., foundation models, physics-informed priors) are likely to shape DLWP performance at subseasonal-to-seasonal scales?

Key findings

• DLWP models incorporate diverse inductive biases across data inputs, forecasting targets, and loss design that drive performance and generalisation.
• Iterative and probabilistic forecasting approaches help manage error accumulation and represent uncertainty in longer forecasts.
• Generative components (GANs, VAEs, dynamic VAEs) contribute to modelling forecast uncertainty and ensemble spread, particularly for long-range predictions.
• High-resolution data and physics-informed inputs (e.g., orography, land-sea masks, solar radiation) are used to encode domain structure and improve skill.
• The authors anticipate a shift toward foundation models trained on large datasets with explicit physics priors to maintain performance at subseasonal-to-seasonal scales.