[Paper Review] Machine Learning Climate Model Dynamics: Offline versus Online Performance

Noah Brenowitz, Brian Henn+6 more|arXiv (Cornell University)|Nov 5, 2020

Meteorological Phenomena and Simulations12 references39 citations

TL;DR

This paper compares random forests and neural networks as parametrizations for moist physics in a 3 km global climate model, showing offline NN accuracy but online instability, while RF remains stable with modest online gains.

ABSTRACT

Climate models are complicated software systems that approximate atmospheric and oceanic fluid mechanics at a coarse spatial resolution. Typical climate forecasts only explicitly resolve processes larger than 100 km and approximate any process occurring below this scale (e.g. thunderstorms) using so-called parametrizations. Machine learning could improve upon the accuracy of some traditional physical parametrizations by learning from so-called global cloud-resolving models. We compare the performance of two machine learning models, random forests (RF) and neural networks (NNs), at parametrizing the aggregate effect of moist physics in a 3 km resolution global simulation with an atmospheric model. The NN outperforms the RF when evaluated offline on a testing dataset. However, when the ML models are coupled to an atmospheric model run at 200 km resolution, the NN-assisted simulation crashes with 7 days, while the RF-assisted simulations remain stable. Both runs produce more accurate weather forecasts than a baseline configuration, but globally averaged climate variables drift over longer timescales.

Motivation & Objective

• Demonstrate ML parametrization feasibility within a more realistic atmospheric model (FV3GFS) at ~3 km resolution.
• Compare offline versus online performance of RF and NN for moist physics parametrizations.
• Assess stability and forecast skill when ML parametrizations are coupled to a coarser 200 km atmospheric model.
• Highlight robustness and drift issues in ML-assisted climate simulations and suggest directions for improvement.

Proposed method

• Train RF and NN to predict moisture and temperature source terms f from locally advanced inputs x_i derived from 3 km FV3GFS data.
• Compute f as the budget residual: f_i = dx_i/dt − g_i, with g_i representing known physics (advection plus clouds-off parametrizations).
• Use nudging to obtain smooth estimates of g_i and x_i from fine-resolution data, with a 3-hour nudging timescale as regularization.
• Features include vertical profiles of temperature and humidity, cosine of solar zenith angle, surface elevation, and land-sea mask.
• RF: max depth 13, 13 trees (batches of 10 timesteps). NN: two layers, 128 nodes each, ReLU,Adam optimizer (lr 0.001).
• Train with MSE loss scaled by inverse output std; batch size 512; 8 epochs; inputs normalized per batch.

Experimental results

Research questions

• RQ1 Can ML-based parametrizations (RF and NN) accurately reproduce moist physics tendencies when trained offline on a high-resolution dataset?
• RQ2 Do offline (testing) accuracies translate into stable online performance when coupled to a coarser atmospheric model?
• RQ3 Which model (RF vs NN) offers better online stability and forecast skill for climate-relevant variables (PW, Z500)?
• RQ4 What are the observed biases and drifts in ML-assisted simulations compared to all-physics baselines?
• RQ5 What implications do offline-online discrepancies have for deploying ML parametrizations in realistic climate models?

Key findings

• Offline, NN outperforms RF in accuracy on the testing dataset (higher R^2).
• Online, NN-assisted simulations crash after ~7 days when coupled to 200 km model, while RF-assisted simulations complete 10-day hindcasts.
• Both ML-assisted runs provide better forecast skill than the all-physics baseline for up to ~5 days, but global PW and Z500 exhibit drift over longer timescales.
• RF tends to be more stable online due to averaging behavior; NN can be unstable due to coupling feedbacks and non-Gaussian inputs.
• NN predictions are globally biased offline (heating) and tend to have less robustness to extreme events; RF remains more robust to extrapolation.