[Paper Review] Ensemble weather forecast post-processing with a flexible probabilistic neural network approach

Peter Mlakar, Janko Merše+1 more|arXiv (Cornell University)|Mar 29, 2023

Meteorological Phenomena and Simulations4 citations

TL;DR

This paper proposes ANET2, a novel neural network-based post-processing method for ensemble weather forecasts that jointly models all locations and lead times using normalizing flows for flexible, exact probabilistic distribution estimation. It achieves state-of-the-art performance on the EUPPBench benchmark by outperforming prior methods, including its predecessor ANET1, through improved distribution modeling and joint inference without per-lead-time or per-location model fitting.

ABSTRACT

Ensemble forecast post-processing is a necessary step in producing accurate probabilistic forecasts. Conventional post-processing methods operate by estimating the parameters of a parametric distribution, frequently on a per-location or per-lead-time basis. We propose a novel, neural network-based method, which produces forecasts for all locations and lead times, jointly. To relax the distributional assumption of many post-processing methods, our approach incorporates normalizing flows as flexible parametric distribution estimators. This enables us to model varying forecast distributions in a mathematically exact way. We demonstrate the effectiveness of our method in the context of the EUPPBench benchmark, where we conduct temperature forecast post-processing for stations in a sub-region of western Europe. We show that our novel method exhibits state-of-the-art performance on the benchmark, outclassing our previous, well-performing entry. Additionally, by providing a detailed comparison of three variants of our novel post-processing method, we elucidate the reasons why our method outperforms per-lead-time-based approaches and approaches with distributional assumptions.

Motivation & Objective

• To address the limitations of conventional post-processing methods that rely on rigid, per-lead-time or per-location parametric distributions with limited expressive power.
• To develop a unified neural network model that performs joint probabilistic forecasting across all locations and lead times, improving generalization and calibration.
• To relax distributional assumptions by employing normalizing flows for mathematically exact, flexible density estimation of temperature forecasts.
• To evaluate the impact of flexible distribution modeling on forecast calibration and predictive performance in a real-world benchmark setting.
• To demonstrate that joint modeling with advanced distribution estimation significantly outperforms standard per-lead-time or parametric approaches.

Proposed method

• ANET2 employs a deep neural network architecture that processes raw ensemble forecasts, station embeddings, and time features to predict distribution parameters in a joint, end-to-end manner across all locations and lead times.
• It uses normalizing flows—specifically, continuous normalizing flows with spline-based transformations—to model complex, non-Gaussian forecast distributions with exact density estimation.
• The model is trained using proper scoring rules (e.g., log-score) to optimize probabilistic forecast quality, ensuring well-calibrated and sharp predictive distributions.
• The method avoids quantile crossing by construction, as normalizing flows provide a monotonic, invertible transformation from a base distribution to the forecast distribution.
• A variant, ANET2_NORM, uses a fixed normal distribution for comparison, while ANET2_BERN uses Bernstein quantile regression, allowing ablation of distributional assumptions.
• The inference pipeline processes the entire D51 test dataset (167,170 forecasts) in under a minute, with ANET2 requiring 36.08 seconds on a GPU.

Figure 1: The neural network architecture of ANET1. The parameter f denotes the number of features in a dense layer. The parameters t and p denote the lead time and the number of per-lead-time parameters required by the normal distribution model. The circular blocks denote the following operations:

Experimental results

Research questions

• RQ1 Does joint modeling across all locations and lead times improve forecast calibration and sharpness compared to per-lead-time or per-station approaches?
• RQ2 Can normalizing flows significantly enhance probabilistic forecast performance by enabling flexible, exact distribution estimation without parametric assumptions?
• RQ3 How do different distribution estimators (normal, Bernstein, normalizing flow) compare in terms of predictive accuracy and computational cost?
• RQ4 To what extent does relaxing distributional assumptions improve forecast quality in real-world meteorological settings?
• RQ5 Is the increased model complexity of normalizing flows justified by measurable gains in forecast skill on standardized benchmarks?

Key findings

• ANET2 achieves state-of-the-art performance on the EUPPBench benchmark, outperforming its predecessor ANET1 across all evaluation metrics, including log-score, CRPS, and reliability diagrams.
• The use of normalizing flows in ANET2 enables exact, non-parametric density estimation, eliminating issues like quantile crossing and improving calibration.
• ANET2 requires 840 parameters for its distribution model, significantly more than ANET2_NORM (42) and ANET2_BERN (273), yet delivers the best predictive performance.
• Despite higher computational cost, ANET2 processes the full D51 test dataset in 36.08 seconds on a GPU, demonstrating practical feasibility.
• The ablation study confirms that joint modeling across locations and lead times is more effective than per-lead-time approaches, especially when combined with flexible distribution estimation.
• ANET2's superior performance is attributed to both its unified architecture and the expressive power of normalizing flows, which better capture complex, non-Gaussian forecast distributions.

Figure 2: The neural network architecture of ANET2. The parameter f denotes the number of features in a dense layer. The parameters t and p denote the lead time and the number of per-lead-time parameters required by the normalizing flow. The circular blocks denote the following operations: M denotes