[Paper Review] Improving medium-range ensemble weather forecasts with hierarchical ensemble transformers

Zied Ben-Bouallegue, Jonathan A. Weyn+4 more|arXiv (Cornell University)|Mar 30, 2023

Meteorological Phenomena and Simulations9 citations

TL;DR

PoET post-processes ensemble forecasts with hierarchical transformers to calibrate each member, improving 2m temperature skill by up to ~20% and precipitation by ~2% globally, outperforming MBM benchmarks and showing strong generalization across model cycles and datasets.

ABSTRACT

Statistical post-processing of global ensemble weather forecasts is revisited by leveraging recent developments in machine learning. Verification of past forecasts is exploited to learn systematic deficiencies of numerical weather predictions in order to boost post-processed forecast performance. Here, we introduce PoET, a post-processing approach based on hierarchical transformers. PoET has 2 major characteristics: 1) the post-processing is applied directly to the ensemble members rather than to a predictive distribution or a functional of it, and 2) the method is ensemble-size agnostic in the sense that the number of ensemble members in training and inference mode can differ. The PoET output is a set of calibrated members that has the same size as the original ensemble but with improved reliability. Performance assessments show that PoET can bring up to 20% improvement in skill globally for 2m temperature and 2% for precipitation forecasts and outperforms the simpler statistical member-by-member method, used here as a competitive benchmark. PoET is also applied to the ENS10 benchmark dataset for ensemble post-processing and provides better results when compared to other deep learning solutions that are evaluated for most parameters. Furthermore, because each ensemble member is calibrated separately, downstream applications should directly benefit from the improvement made on the ensemble forecast with post-processing.

Motivation & Objective

• Motivate statistical post-processing to correct systematic forecast errors in NWP ensembles.
• Develop a transformer-based post-processing method that operates directly on ensemble members.
• Ensure the method is ensemble-size agnostic and scalable to different spatial resolutions.
• Benchmark PoET against MBM and ENS-10 to assess relative performance across variables.

Proposed method

• Introduce PoET, a post-processing approach based on hierarchical transformers.
• Apply the transformer along the ensemble dimension to recalibrate each ensemble member.
• Embed PoET in a U-net architecture to manage memory and support high-resolution global grids.
• Utilize a residual transformer block with layer normalization to correct forecast errors.
• Train on ECMWF reforecasts and reanalysis (ERA5) with lead times up to 96 hours; test on 2021 operational data.
• Compare PoET to MBM and assess performance on 2m temperature and precipitation; adapt loss (kCRPS for precipitation).

Figure 1: Schematic of the transformer U-net architecture of PoET. The inset on the lower right is adapted from Finn ( 2021 ) , used with permission.

Experimental results

Research questions

• RQ1 Can PoET meaningfully improve the reliability and skill of global 2m temperature and precipitation forecasts relative to raw ensembles?
• RQ2 How does PoET compare with the MBM statistical post-processing method across lead times up to 96 hours?
• RQ3 Is PoET ensemble-size agnostic, maintaining performance when training and inference ensembles differ in size?
• RQ4 Does PoET generalize across different model cycles and to ENS-10 benchmark datasets?
• RQ5 What is the spatial pattern of PoET’s improvements and any regional degradation?

Key findings

• PoET yields global skill improvements of up to ~20% for 2m temperature and ~2% for precipitation over the raw ensemble.
• PoET generally outperforms MBM on 2m temperature across lead times and regions.
• On ENS-10, PoET provides significant improvements over previous baselines, notably for 2m temperature.
• For precipitation, gains are smaller and regionally variable, with better performance in the tropics.
• PoET can operate with different ensemble sizes between training and inference, confirming ensemble-size agnosticism.
• Calibrating each ensemble member separately via PoET yields better alignment with ERA5 and supports downstream applications.

Figure 2: Relative changes by PoET to a single date of the 2m temperature forecast at day 2, valid on 6 January 2021. The left column shows the first 3 raw ensemble members. The first row shows the first 3 PoET-corrected ensemble members. Other entries show the differences between raw and PoET-corre