[Paper Review] Decision-oriented benchmarking to transform AI weather forecast access: Application to the Indian monsoon

Rajat Masiwal, Colin Aitken+10 more|arXiv (Cornell University)|Feb 3, 2026

Climate variability and models0 citations

TL;DR

The paper presents a decision-oriented benchmarking framework that ties meteorology, AI-based weather prediction, and social science to assess open-source AI weather models for local Indian monsoon onset and demonstrates dissemination to 38 million farmers.

ABSTRACT

Artificial intelligence weather prediction (AIWP) models now often outperform traditional physics-based models on common metrics while requiring orders-of-magnitude less computing resources and time. Open-access AIWP models thus hold promise as transformational tools for helping low- and middle-income populations make decisions in the face of high-impact weather shocks. Yet, current approaches to evaluating AIWP models focus mainly on aggregated meteorological metrics without considering local stakeholders' needs in decision-oriented, operational frameworks. Here, we introduce such a framework that connects meteorology, AI, and social sciences. As an example, we apply it to the 150-year-old problem of Indian monsoon forecasting, focusing on benefits to rain-fed agriculture, which is highly susceptible to climate change. AIWP models skillfully predict an agriculturally relevant onset index at regional scales weeks in advance when evaluated out-of-sample using deterministic and probabilistic metrics. This framework informed a government-led effort in 2025 to send 38 million Indian farmers AI-based monsoon onset forecasts, which captured an unusual weeks-long pause in monsoon progression. This decision-oriented benchmarking framework provides a key component of a blueprint for harnessing the power of AIWP models to help large vulnerable populations adapt to weather shocks in the face of climate variability and change.

Motivation & Objective

• Address the gap between meteorological benchmarking and operational, decision-oriented dissemination in LMICs.
• Develop a framework that links climate science, AI, and development economics for practical impact.
• Evaluate local monsoon onset forecasts using AIWP models in an agriculturally relevant, regional context.
• Inform large-scale forecast dissemination and model selection for decision-support in farming.
• Demonstrate how benchmarking can guide government-led deployment of AI forecasts.

Proposed method

• Define a locally relevant monsoon onset index compatible with operational constraints.
• Hindcast and evaluate six AIWP models and one NWP model against IMD observations across multiple periods and regions.
• Use both deterministic (MAE, MR, FAR) and probabilistic (BSS, RPSS, AUC) metrics to assess skill.
• Compare against a climatological baseline and use open-source models to enable rapid hindcast generation.
• Demonstrate model blending and selection for dissemination in a real-world program.
• Highlight the need for post-processing and calibration for actionable probabilistic forecasts.

Figure 1: Decision-oriented operational benchmarking framework: the Indian monsoon onset example . Panels (a)–(b) present an operationally oriented benchmarking framework for global AIWP models using locally relevant, decision-oriented metrics. One example is an agriculturally relevant metric of the

Experimental results

Research questions

• RQ1 Can open-source AI weather prediction models skillfully forecast local monsoon onset weeks in advance compared with a climatological baseline?
• RQ2 Do probabilistic forecasts from AIWP models provide added value for decision-making in agriculture?
• RQ3 How do model performances vary spatially within the core monsoon zone and across different lead times?
• RQ4 Can a decision-oriented benchmarking framework inform large-scale dissemination of forecasts to farmers in India?
• RQ5 What lessons emerge for operational deployment and calibration of AIWP forecasts in LMIC contexts?

Key findings

• Most AIWP and NWP models outperform climatology for 1–15 day lead times in predicting local monsoon onset in the core monsoon zone (CMZ) on deterministic metrics like MAE and MR.
• Model skill generally degrades toward 16–30 day lead times, with some models maintaining useful skill for certain metrics.
• Probabilistic forecasts show skill for some models up to 15–30 days, enabling ensemble-based decision support, though many forecasts are overconfident and require calibration.
• An operational 2025 dissemination effort in India used two AIWP models (AIFS and NGCM) to provide calibrated probabilistic onset forecasts to about 38 million farmers, demonstrating real-world utility.
• The decision-oriented framework reveals the value of local, action-focused indices and multi-metric evaluation in guiding model selection and dissemination.
• The framework underscores the need for post-processing, calibration, and tailored messaging to translate forecast skill into actionable farmer decisions.

Figure 2: Deterministic forecast skill of the models in the medium-range (1-15 day) and subseasonal (16-30 day) timescales across two analysis periods. (a) Forecast skill of local rainfall-based monsoon onset (defined in Methods), averaged over the CMZ (outlined in panel (b)) at 1-15 day (left) and