[论文解读] ClimateLearn: Benchmarking Machine Learning for Weather and Climate Modeling

Tung Nguyen, Jason Jewik+3 more|arXiv (Cornell University)|Jul 4, 2023

Meteorological Phenomena and Simulations被引用 9

一句话总结

ClimateLearn 是一个开源的 PyTorch 库，提供天气与气候任务的端到端 ML 基准测试，包含数据集、模型和评估工具以提高可重复性。

ABSTRACT

Modeling weather and climate is an essential endeavor to understand the near- and long-term impacts of climate change, as well as inform technology and policymaking for adaptation and mitigation efforts. In recent years, there has been a surging interest in applying data-driven methods based on machine learning for solving core problems such as weather forecasting and climate downscaling. Despite promising results, much of this progress has been impaired due to the lack of large-scale, open-source efforts for reproducibility, resulting in the use of inconsistent or underspecified datasets, training setups, and evaluations by both domain scientists and artificial intelligence researchers. We introduce ClimateLearn, an open-source PyTorch library that vastly simplifies the training and evaluation of machine learning models for data-driven climate science. ClimateLearn consists of holistic pipelines for dataset processing (e.g., ERA5, CMIP6, PRISM), implementation of state-of-the-art deep learning models (e.g., Transformers, ResNets), and quantitative and qualitative evaluation for standard weather and climate modeling tasks. We supplement these functionalities with extensive documentation, contribution guides, and quickstart tutorials to expand access and promote community growth. We have also performed comprehensive forecasting and downscaling experiments to showcase the capabilities and key features of our library. To our knowledge, ClimateLearn is the first large-scale, open-source effort for bridging research in weather and climate modeling with modern machine learning systems. Our library is available publicly at https://github.com/aditya-grover/climate-learn.

研究动机与目标

• 通过标准化数据集、模型和评估协议，促进天气与气候的可重复 ML 研究。
• 提供涵盖预测、尺度下采样和气候投影的端到端 ML 流程。
• 提供传统基线和鲁棒评估与可视化工具的最新 DL 模型。
• 促进跨数据集的鲁棒性与极端事件基准测试，以检测模型泛化能力。

提出的方法

• 开源 PyTorch 库，包含任务、数据集、模型和评估四个组成部分。
• 支持 ERA5、CMIP6 和 PRISM 数据，具多种分辨率和网格配置。
• 实现传统基线（气候学、保持性、线性回归）和 DL 模型（ResNet、U-Net、ViT），以及预训练/可导入模型。
• 提供确定性和概率预测、尺度下采样与气候投影的评估指标，包括经纬度加权变体与可视化诊断。
• 支持连续、直接和迭代预测分析，以比较训练策略与提前期性能。
• 包含丰富的文档、教程和可重复的基准设置。

Figure 2: Performance on forecasting three variables at different lead times. Solid lines are deep learning methods, dashed lines are simple baselines, and the dotted line is the physics-based model. Lower RMSE and higher ACC indicate better performance.

实验结果

研究问题

• RQ1 深度学习模型与传统基线在标准数据集上的天气预测、尺度下采样和气候投影表现是否存在差异？
• RQ2 预测策略（直接、连续、迭代）对预测性能与计算成本有何影响？
• RQ3 跨数据集（ERA5 与 CMIP6）对极端天气事件和分布偏移的鲁棒性如何？
• RQ4 DL 模型能否在数据集之间实现有效迁移（在 CMIP6/ERA5 上训练并应用于其他任务）以及对尺度下采样任务（从 ERA5 到 PRISM）的迁移？
• RQ5 ClimateLearn 在评估不确定性和提供诊断方面具有什么定性与定量能力？

主要发现

• 深度学习方法（尤其是 ResNet）通常在预测方面优于气候学与保持性基线，但在某些情境下可能落后于物理基线 IFS。
• 连续预测在较长提前期可达到或超过直接预测，而迭代预测常因误差累积而表现不佳。
• 极端 ERA5 情况下，深度学习与保持性表现优于 ERA5 基线；气候学在极端条件下性能下降。
• 跨数据集实验表明在 CMIP6 上训练可提升对 ERA5 的评估分数，显示对分布偏移具有一定鲁棒性。
• 使用 DL 方法进行尺度下采样在 RMSE 上优于最近邻/双线性插值，但在某些情境下模型可能存在负偏差；跨数据集测试中 DL 方法显示更高的皮尔逊相关。

Figure 3: Comparison of direct, continuous, and iterative forecasting with ResNet architecture.