[Paper Review] On some limitations of data-driven weather forecasting models

Massimo Bonavita|arXiv (Cornell University)|Sep 15, 2023

Meteorological Phenomena and Simulations8 citations

TL;DR

The paper analyzes fidelity and physical consistency of data-driven weather forecasts (exemplified by Pangu-Weather) and argues that while ML models can add value for specific applications, they may lack fidelity and physical consistency compared to physics-based models, influencing traditional forecast skill interpretations.

ABSTRACT

As in many other areas of engineering and applied science, Machine Learning (ML) is having a profound impact in the domain of Weather and Climate Prediction. A very recent development in this area has been the emergence of fully data-driven ML prediction models which routinely claim superior performance to that of traditional physics-based models. In this work, we examine some aspects of the forecasts produced by an exemplar of the current generation of ML models, Pangu-Weather, with a focus on the fidelity and physical consistency of those forecasts and how these characteristics relate to perceived forecast performance. The main conclusion is that Pangu-Weather forecasts, and possibly those of similar ML models, do not have the fidelity and physical consistency of physics-based models and their advantage in accuracy on traditional deterministic metrics of forecast skill can be at least partly attributed to these peculiarities. Balancing forecast skill and physical consistency of ML-driven predictions will be an important consideration for future ML models. However, and similarly to other modern post-processing technologies, the current ML models appear to be already able to add value to standard NWP output for specific forecast applications and combined with their extremely low computational cost during deployment, are set to provide an additional, useful source of forecast information. .

Motivation & Objective

• Motivate evaluation of data-driven weather forecasts beyond traditional skill metrics.
• Assess fidelity of ML forecasts relative to physics-based models.
• Examine physical consistency of data-driven predictions and their implications for trusted forecasting.
• Identify how ML models can add value for specific applications despite limitations.

Proposed method

• Examine forecasts produced by an exemplar data-driven model (Pangu-Weather).
• Assess fidelity and physical consistency of ML forecasts in the context of weather prediction.
• Compare ML forecast characteristics to physics-based model expectations.
• Discuss how low computational cost and post-processing influence forecast usefulness.

Experimental results

Research questions

• RQ1 Do data-driven weather forecasts exhibit fidelity comparable to physics-based models?
• RQ2 Are the forecasts physically consistent with established atmospheric dynamics and conservation laws?
• RQ3 How does the apparent forecast skill of ML models relate to their fidelity and physical consistency?
• RQ4 Can ML models provide valuable added forecast information for specific applications despite limitations?

Key findings

• ML forecasts may lack fidelity and physical consistency compared to physics-based models.
• The advantage of ML models on traditional deterministic skill metrics may partly reflect these peculiarities rather than true physical accuracy.
• ML forecasts can still add value for specific forecast applications.
• The extremely low computational cost of ML deployment supports their use as an additional source of forecast information when combined with standard NWP outputs.