[Paper Review] CLLMate: A Multimodal Benchmark for Weather and Climate Events Forecasting

Haobo Li, Zhaowei Wang+4 more|arXiv (Cornell University)|Sep 27, 2024

Advanced Computational Techniques and Applications4 citations

TL;DR

This paper introduces CLLMate, a multimodal large language model that forecasts open-set weather and climate events by aligning meteorological raster data with textual event descriptions using a knowledge graph constructed from 41,000 environmental news articles. The model outperforms baselines and existing multimodal LLMs, demonstrating the feasibility of LLM-based multimodal alignment for comprehensive environmental risk forecasting.

ABSTRACT

Forecasting weather and climate events is crucial for making appropriate measures to mitigate environmental hazards and minimize losses. However, existing environmental forecasting research focuses narrowly on predicting numerical meteorological variables (e.g., temperature), neglecting the translation of these variables into actionable textual narratives of events and their consequences. To bridge this gap, we proposed Weather and Climate Event Forecasting (WCEF), a new task that leverages numerical meteorological raster data and textual event data to predict weather and climate events. This task is challenging to accomplish due to difficulties in aligning multimodal data and the lack of supervised datasets. To address these challenges, we present CLLMate, the first multimodal dataset for WCEF, using 26,156 environmental news articles aligned with ERA5 reanalysis data. We systematically benchmark 23 existing MLLMs on CLLMate, including closed-source, open-source, and our fine-tuned models. Our experiments reveal the advantages and limitations of existing MLLMs and the value of CLLMate for the training and benchmarking of the WCEF task.

Motivation & Objective

• Address the gap in forecasting open-set weather and climate events, which are not captured by traditional numerical variable predictions.
• Overcome the challenge of aligning multimodal data—specifically meteorological raster data and textual event descriptions—due to lack of supervised datasets.
• Construct a large-scale, supervised multimodal dataset by linking meteorological data with events extracted from environmental news using LLMs.
• Develop a multimodal LLM, CLLMate, capable of forecasting diverse, real-world weather and climate events beyond predefined closed-set categories.
• Contribute a practical solution to the WCRP’s "My Climate Risk" lighthouse initiative for regional climate risk forecasting.

Proposed method

• Use a large language model (LLM) to extract event-related information (e.g., type, location, time, impact) from a corpus of 41,000 environment-focused news articles.
• Construct a knowledge graph from the extracted event data to represent relationships between events and their contextual attributes.
• Map historical meteorological raster data (e.g., temperature, pressure, wind) to events in the knowledge graph using spatio-temporal alignment.
• Create a supervised multimodal instruction dataset by pairing meteorological sequences with corresponding textual event descriptions for training.
• Train CLLMate, a multimodal LLM that uses a vision encoder to process meteorological raster data and projects the features into the LLM space for joint reasoning.
• Apply instruction tuning with task-formatted prompts to align the model’s outputs with the WCEF task, using synthetic data generated via strong LLMs.

Figure 1. The multipart LLM prompt for extracting weather and climate events and their relationships.

Experimental results

Research questions

• RQ1 Can LLMs effectively extract and structure event information from unstructured environmental news articles to support multimodal forecasting?
• RQ2 How can meteorological raster data be reliably aligned with textual weather and climate events to form a supervised dataset?
• RQ3 Can a multimodal LLM like CLLMate outperform traditional baselines and existing multimodal LLMs in forecasting open-set weather and climate events?
• RQ4 What is the impact of multimodal alignment between numerical meteorological data and textual event descriptions on forecasting accuracy?
• RQ5 To what extent can LLM-based models generalize to complex, real-world environmental risk scenarios beyond predefined event types?

Key findings

• CLLMate achieves superior performance over all baseline models and existing multimodal LLMs in forecasting weather and climate events.
• The constructed dataset, derived from 41,000 news articles and aligned with meteorological data, is the largest and most novel supervised dataset for WCEF task tuning.
• The knowledge graph-based event extraction from news articles enables effective identification of diverse, real-world events, including secondary and tertiary impacts.
• Multimodal alignment using vision encoders for raster data and LLMs for text significantly improves forecasting accuracy for open-set events.
• Instruction tuning with synthetic, task-formatted data enhances the model’s ability to generate accurate, context-aware forecasts.
• CLLMate demonstrates strong generalization to complex, non-closed-set events such as infrastructure failures due to extreme weather, which are often missed by traditional methods.

Figure 2. The knowledge graph of weather and climate events and its natural language description.