[Paper Review] Comparison of Model Output Statistics and Neural Networks to Postprocess Wind Gusts

Cristina Primo, Benedikt Schulz+2 more|arXiv (Cornell University)|Jan 22, 2024

Meteorological Phenomena and Simulations4 citations

TL;DR

This study compares Model Output Statistics (MOS) using linear and logistic regression with advanced neural network methods—specifically distributional regression networks (DRN) and Bernstein quantile networks (BQN)—for postprocessing wind gust forecasts. It finds that neural networks, particularly with extended training data including model changes, significantly improve forecast calibration and reliability over traditional MOS, offering a strong case for operational adoption in weather services.

ABSTRACT

Wind gust prediction plays an important role in warning strategies of national meteorological services due to the high impact of its extreme values. However, forecasting wind gusts is challenging because they are influenced by small-scale processes and local characteristics. To account for the different sources of uncertainty, meteorological centers run ensembles of forecasts and derive probabilities of wind gusts exceeding a threshold. These probabilities often exhibit systematic errors and require postprocessing. Model Output Statistics (MOS) is a common operational postprocessing technique, although more modern methods such as neural network-bases approaches have shown promising results in research studies. The transition from research to operations requires an exhaustive comparison of both techniques. Taking a first step into this direction, our study presents a comparison of a postprocessing technique based on linear and logistic regression approaches with different neural network methods proposed in the literature to improve wind gust predictions, specifically distributional regression networks and Bernstein quantile networks. We further contribute to investigating optimal design choices for neural network-based postprocessing methods regarding changes of the numerical model in the training period, the use of persistence predictors, and the temporal composition of training datasets. The performance of the different techniques is compared in terms of calibration, accuracy, reliability and resolution based on case studies of wind gust forecasts from the operational weather model of the German weather service and observations from 170 weather stations.

Motivation & Objective

• To evaluate the performance of neural network-based postprocessing methods (DRN, BQN) against traditional Model Output Statistics (MOS) for wind gust forecasts.
• To assess the impact of model changes during training—such as updates to the ICON model and data assimilation (KENDA)—on neural network forecast quality.
• To investigate optimal design choices for neural network postprocessing, including use of persistence predictors and temporal composition of training datasets.
• To provide a fair, empirical comparison under realistic operational conditions using real-world data from 170 German weather stations.
• To support the transition from research to operations by evaluating the feasibility and benefits of adopting neural networks in national meteorological services.

Proposed method

• Employs a two-step postprocessing pipeline: first, individual model forecasts (ICON and IFS) are postprocessed using MOS or neural networks; second, results are optimally combined into a unified probabilistic forecast.
• Uses distributional regression networks (DRN) and Bernstein quantile networks (BQN) to model the full predictive distribution of wind gusts, enabling probabilistic forecasts at any threshold.
• Trains neural networks on extended datasets that include periods with model changes (e.g., new ICON model, KENDA data assimilation), using binary indicators to account for such shifts.
• Applies persistence predictors (e.g., recent observed gusts) to improve temporal consistency and forecast skill, especially in rapidly evolving weather conditions.
• Uses score cards summarizing verification metrics (e.g., CRPS, reliability, resolution) to enable direct, comprehensive comparison between MOS and neural network methods.
• Evaluates time consistency and computational efficiency to assess suitability for operational use, particularly in real-time warning systems.

Experimental results

Research questions

• RQ1 How do neural network-based postprocessing methods (DRN and BQN) compare to traditional MOS in terms of forecast calibration, accuracy, and reliability for wind gusts?
• RQ2 What is the impact of including model changes (e.g., new ICON model, KENDA data assimilation) in the training data on the performance of neural network postprocessing models?
• RQ3 How do different design choices—such as the use of persistence predictors and temporal composition of training data—affect the predictive performance of neural network postprocessing?
• RQ4 Can neural networks trained on longer, heterogeneous training datasets (including model changes) outperform those trained on homogeneous data in operational wind gust forecasting?
• RQ5 What are the practical implications of using neural networks over MOS in operational weather services, particularly regarding time consistency and computational efficiency?

Key findings

• Neural network-based postprocessing methods, especially DRN and BQN, significantly improve forecast calibration and reliability compared to traditional MOS, particularly in capturing extreme wind gust events.
• Training neural networks on longer datasets that include model changes (e.g., new ICON model, KENDA data assimilation) leads to better forecast performance than training on homogeneous data, even when model changes occur.
• The inclusion of persistence predictors enhances forecast accuracy and temporal consistency, especially in short-term wind gust predictions.
• DRN and BQN provide full predictive distributions, enabling consistent probabilistic forecasts across all thresholds, unlike MOS which typically focuses on threshold exceedance probabilities.
• Despite the complexity of neural networks, they maintain time consistency and computational feasibility suitable for operational use when properly designed.
• The study demonstrates that fair, direct comparisons between operational MOS and modern neural network methods are feasible and reveal clear advantages of neural networks in key verification metrics such as CRPS and reliability.