[Paper Review] MMST-ViT: Climate Change-aware Crop Yield Prediction via Multi-Modal Spatial-Temporal Vision Transformer

Fudong Lin, Summer Crawford+15 more|arXiv (Cornell University)|Sep 16, 2023

Remote Sensing in Agriculture4 citations

TL;DR

This paper proposes MMST-ViT, a Vision Transformer-based model that integrates multi-modal remote sensing and meteorological data to predict county-level crop yields in the U.S., accounting for both short-term weather variations and long-term climate change. It achieves state-of-the-art performance through a novel multi-modal contrastive learning pre-training strategy and dedicated spatial-temporal attention mechanisms.

ABSTRACT

Precise crop yield prediction provides valuable information for agricultural planning and decision-making processes. However, timely predicting crop yields remains challenging as crop growth is sensitive to growing season weather variation and climate change. In this work, we develop a deep learning-based solution, namely Multi-Modal Spatial-Temporal Vision Transformer (MMST-ViT), for predicting crop yields at the county level across the United States, by considering the effects of short-term meteorological variations during the growing season and the long-term climate change on crops. Specifically, our MMST-ViT consists of a Multi-Modal Transformer, a Spatial Transformer, and a Temporal Transformer. The Multi-Modal Transformer leverages both visual remote sensing data and short-term meteorological data for modeling the effect of growing season weather variations on crop growth. The Spatial Transformer learns the high-resolution spatial dependency among counties for accurate agricultural tracking. The Temporal Transformer captures the long-range temporal dependency for learning the impact of long-term climate change on crops. Meanwhile, we also devise a novel multi-modal contrastive learning technique to pre-train our model without extensive human supervision. Hence, our MMST-ViT captures the impacts of both short-term weather variations and long-term climate change on crops by leveraging both satellite images and meteorological data. We have conducted extensive experiments on over 200 counties in the United States, with the experimental results exhibiting that our MMST-ViT outperforms its counterparts under three performance metrics of interest.

Motivation & Objective

• Address the challenge of accurate county-level crop yield prediction under dynamic weather and long-term climate change.
• Overcome limitations of existing models that focus on either remote sensing or meteorological data, but not both.
• Develop a deep learning framework that captures both short-term weather impacts and long-term climate trends on crop yields.
• Enable self-supervised pre-training using multi-modal data to reduce reliance on human-annotated labels.
• Improve prediction accuracy by integrating spatial dependencies across counties and temporal dependencies over historical climate data.

Proposed method

• Proposes a Multi-Modal Spatial-Temporal Vision Transformer (MMST-ViT) with three specialized attention heads: Multi-Modal, Spatial, and Temporal.
• Employs Multi-Head Attention (MHA) in the Multi-Modal Transformer to jointly process Sentinel-2 satellite imagery and HRRR meteorological data during the growing season.
• Uses a Spatial Transformer with MHA to model high-resolution spatial dependencies among neighboring counties for precise agricultural tracking.
• Utilizes a Temporal Transformer with MHA to capture long-range temporal dependencies in historical meteorological data for climate change impact modeling.
• Introduces a novel multi-modal contrastive learning pre-training technique that leverages both visual and meteorological data without human annotations.
• Applies average pooling as a baseline for comparison in ablation studies to evaluate the necessity of attention-based global representation learning.

Experimental results

Research questions

• RQ1 Can a Vision Transformer-based model effectively integrate multi-modal remote sensing and meteorological data for crop yield prediction?
• RQ2 How does the inclusion of long-term historical climate data improve the accuracy of crop yield predictions compared to short-term weather data alone?
• RQ3 To what extent do spatial dependencies among counties enhance prediction performance at the county level?
• RQ4 Does multi-modal self-supervised pre-training significantly improve model generalization and reduce overfitting in low-data regimes?
• RQ5 How do the individual components (Multi-Modal, Spatial, Temporal Transformers) contribute to overall prediction performance?

Key findings

• MMST-ViT achieves a correlation (Corr) of 0.900 for corn and 0.918 for soybean, significantly outperforming models without pre-training (Corr: 0.857 and 0.875) and those pre-trained with SimCLR (Corr: 0.857 and 0.876).
• The removal of the Temporal Transformer reduces Corr by 0.080 for corn and 0.069 for soybean, demonstrating its critical role in modeling long-term climate change effects.
• Masking short-term meteorological data degrades Corr by 0.096 for soybean, highlighting the importance of real-time weather data in crop growth modeling.
• The model’s multi-modal contrastive pre-training reduces RMSE by 2.7 (corn) and 1.2 (soybean) compared to training without pre-training, indicating strong generalization benefits.
• The absence of the Multi-Modal Transformer (via masking) leads to a Corr drop of 0.073 for soybean, confirming that joint modeling of imagery and weather is essential.
• Pre-training with SimCLR yields only a 0.001 Corr improvement for soybean, underscoring the necessity of a multi-modal pre-training approach over single-modality methods.