[Paper Review] Deep Uncertainty Quantification: A Machine Learning Approach for Weather Forecasting

Bin Wang, Jie Lü+5 more|arXiv (Cornell University)|Dec 22, 2018

Hydrological Forecasting Using AI29 references21 citations

TL;DR

This paper proposes Deep Uncertainty Quantification (DUQ), a novel deep learning framework that jointly performs point forecasting and uncertainty quantification in weather prediction by leveraging historical observations and numerical weather prediction (NWP) data through an end-to-end trainable model. The method introduces a negative log-likelihood error (NLE) loss function that significantly improves generalization over MSE and MAE, achieving a 47.76% accuracy improvement over NWP on a Beijing weather dataset, with state-of-the-art performance and open-sourced benchmarks.

ABSTRACT

Weather forecasting is usually solved through numerical weather prediction (NWP), which can sometimes lead to unsatisfactory performance due to inappropriate setting of the initial states. In this paper, we design a data-driven method augmented by an effective information fusion mechanism to learn from historical data that incorporates prior knowledge from NWP. We cast the weather forecasting problem as an end-to-end deep learning problem and solve it by proposing a novel negative log-likelihood error (NLE) loss function. A notable advantage of our proposed method is that it simultaneously implements single-value forecasting and uncertainty quantification, which we refer to as deep uncertainty quantification (DUQ). Efficient deep ensemble strategies are also explored to further improve performance. This new approach was evaluated on a public dataset collected from weather stations in Beijing, China. Experimental results demonstrate that the proposed NLE loss significantly improves generalization compared to mean squared error (MSE) loss and mean absolute error (MAE) loss. Compared with NWP, this approach significantly improves accuracy by 47.76%, which is a state-of-the-art result on this benchmark dataset. The preliminary version of the proposed method won 2nd place in an online competition for daily weather forecasting.

Motivation & Objective

• To address the limitations of numerical weather prediction (NWP) due to sensitivity to initial conditions and instability in differential equations.
• To develop a data-driven method that effectively fuses historical observations with NWP outputs to improve forecasting accuracy.
• To simultaneously achieve high-accuracy point forecasting and reliable uncertainty quantification (prediction intervals) in time series regression.
• To design a differentiable loss function compatible with standard deep learning frameworks that enhances model generalization.
• To establish a publicly available benchmark for weather forecasting using machine learning.

Proposed method

• Proposes a deep neural network architecture that ingests both historical weather observations and NWP model outputs through an effective information fusion mechanism.
• Introduces a novel negative log-likelihood error (NLE) loss function to train the model end-to-end, enabling simultaneous point estimation and uncertainty quantification.
• The NLE loss is differentiable and compatible with PyTorch and TensorFlow, allowing backpropagation-based optimization without requiring Bayesian inference.
• Employs efficient deep ensemble strategies—training multiple models with diverse initializations and data splits—to improve robustness and predictive performance.
• Uses a sequence-to-sequence framework to predict multiple meteorological variables (temperature, humidity, wind speed) across multiple weather stations and future time steps.
• Applies a 90% prediction interval (PI) to evaluate uncertainty calibration, with coverage measured via PICP (Prediction Interval Coverage Probability).

Experimental results

Research questions

• RQ1 Can a deep learning model effectively fuse historical observations and NWP outputs to improve weather forecasting accuracy?
• RQ2 Does the proposed negative log-likelihood error (NLE) loss function lead to better generalization than standard losses like MSE and MAE in weather forecasting?
• RQ3 Can a single deep learning model simultaneously produce accurate point forecasts and reliable prediction intervals?
• RQ4 How does the performance of the proposed DUQ method compare to traditional NWP and other machine learning baselines on real-world weather data?
• RQ5 To what extent do ensemble strategies enhance the accuracy and robustness of the forecasting system?

Key findings

• The proposed NLE loss function significantly improves generalization in point forecasting compared to MSE and MAE, a phenomenon not previously reported in the literature.
• DUQ outperforms NWP by 47.76% in accuracy on the Beijing weather dataset, achieving state-of-the-art performance on this benchmark.
• The DUQ 300-300 ensemble model achieved the lowest average RMSE of 3.3669, significantly outperforming all baselines including Seq2Seq with MSE and MAE losses.
• The prediction intervals generated by DUQ maintain high coverage, with an average PICP of 0.9249 for temperature, 0.9099 for relative humidity, and 0.9519 for wind speed, indicating well-calibrated uncertainty estimates.
• The deep ensemble strategy significantly improves performance, with the DUQ 300-300 model achieving the best results across all metrics, and the ensemble convergence requiring fewer iterations (2900) than baseline models.
• Ablation studies show that removing either NWP or observational data leads to a substantial performance drop, confirming the importance of the fused information architecture.