[논문 리뷰] ClimateLearn: Benchmarking Machine Learning for Weather and Climate Modeling

Tung Nguyen, Jason Jewik외 3명|arXiv (Cornell University)|2023. 07. 04.

Meteorological Phenomena and Simulations인용 수 9

한 줄 요약

ClimateLearn은 weather와 climate 작업(예측, 다운스케일링, 기후 예측)에 대한 end-to-end ML 벤치마크를 제공하는 오픈 소스 PyTorch 라이브러리로, 재현성을 개선하기 위한 데이터셋, 모델 및 평가 도구를 제공합니다.

ABSTRACT

Modeling weather and climate is an essential endeavor to understand the near- and long-term impacts of climate change, as well as inform technology and policymaking for adaptation and mitigation efforts. In recent years, there has been a surging interest in applying data-driven methods based on machine learning for solving core problems such as weather forecasting and climate downscaling. Despite promising results, much of this progress has been impaired due to the lack of large-scale, open-source efforts for reproducibility, resulting in the use of inconsistent or underspecified datasets, training setups, and evaluations by both domain scientists and artificial intelligence researchers. We introduce ClimateLearn, an open-source PyTorch library that vastly simplifies the training and evaluation of machine learning models for data-driven climate science. ClimateLearn consists of holistic pipelines for dataset processing (e.g., ERA5, CMIP6, PRISM), implementation of state-of-the-art deep learning models (e.g., Transformers, ResNets), and quantitative and qualitative evaluation for standard weather and climate modeling tasks. We supplement these functionalities with extensive documentation, contribution guides, and quickstart tutorials to expand access and promote community growth. We have also performed comprehensive forecasting and downscaling experiments to showcase the capabilities and key features of our library. To our knowledge, ClimateLearn is the first large-scale, open-source effort for bridging research in weather and climate modeling with modern machine learning systems. Our library is available publicly at https://github.com/aditya-grover/climate-learn.

연구 동기 및 목표

• Promote reproducible ML research for weather and climate by standardizing datasets, models, and evaluation protocols.
• Provide end-to-end ML pipelines covering forecasting, downscaling, and climate projection.
• Offer traditional baselines and state-of-the-art DL models with robust evaluation and visualization utilities.
• Facilitate cross-dataset robustness and extreme-event benchmarking to test model generalization.

제안 방법

• Open-source PyTorch library with four components: tasks, datasets, models, and evaluations.
• Supports ERA5, CMIP6, and PRISM data with multiple resolutions and grid configurations.
• Implements traditional baselines (climatology, persistence, linear regression) and DL models (ResNet, U-Net, ViT) plus pretrained/importable models.
• Provides metrics for deterministic and probabilistic forecasting, downscaling, and climate projections, including latitude-weighted variants and visual diagnostics.
• Enables continuous, direct, and iterative forecasting analyses to compare training strategies and lead-time performance.
• Includes extensive documentation, tutorials, and reproducible benchmark setup.

Figure 2: Performance on forecasting three variables at different lead times. Solid lines are deep learning methods, dashed lines are simple baselines, and the dotted line is the physics-based model. Lower RMSE and higher ACC indicate better performance.

실험 결과

연구 질문

• RQ1 How do deep learning models compare to traditional baselines for weather forecasting, downscaling, and climate projection across standard datasets?
• RQ2 What is the impact of forecasting strategy (direct vs continuous vs iterative) on predictive performance and computation?
• RQ3 How robust are ML models to extreme weather events and distributional shifts across datasets (ERA5 vs CMIP6)?
• RQ4 Can DL models effectively transfer across datasets (training on CMIP6/ERA5) and to downscaling tasks (ERA5 to PRISM)?
• RQ5 What are the qualitative and quantitative capabilities of ClimateLearn in evaluating uncertainty and providing diagnostics?

주요 결과

• Deep learning methods (especially ResNet) generally outperform climatology and persistence in forecasting but may lag behind physics-based IFS in some settings.
• Continuous forecasting can match or exceed direct forecasting at longer lead times, while iterative forecasting often underperforms due to error accumulation.
• Extreme-ERA5 results show deep learning and persistence outperforming ERA5 baselines; climatology degrades under extreme conditions.
• Cross-dataset experiments show training on CMIP6 can improve ERA5 evaluation scores, indicating some robustness to distribution shifts.
• Downscaling with DL methods surpasses nearest/bilinear interpolation on RMSE, though models may exhibit negative bias in some settings; DL methods show higher Pearson correlations in cross-dataset tests.

Figure 3: Comparison of direct, continuous, and iterative forecasting with ResNet architecture.