[논문 리뷰] Multi-modal learning for geospatial vegetation forecasting

Vitus Benson, Claire Nicolle Robin외 8명|arXiv (Cornell University)|2023. 03. 28.

Remote Sensing in Agriculture인용 수 22

한 줄 요약

The paper extends meteo-guided video prediction to forecast high-resolution vegetation (NDVI) over Europe by conditioning on past/future weather and elevation, using extended EarthNet2021 data and four weather-guided models, achieving state-of-the-art performance for continental-scale vegetation forecasting.

ABSTRACT

The innovative application of precise geospatial vegetation forecasting holds immense potential across diverse sectors, including agriculture, forestry, humanitarian aid, and carbon accounting. To leverage the vast availability of satellite imagery for this task, various works have applied deep neural networks for predicting multispectral images in photorealistic quality. However, the important area of vegetation dynamics has not been thoroughly explored. Our study breaks new ground by introducing GreenEarthNet, the first dataset specifically designed for high-resolution vegetation forecasting, and Contextformer, a novel deep learning approach for predicting vegetation greenness from Sentinel 2 satellite images with fine resolution across Europe. Our multi-modal transformer model Contextformer leverages spatial context through a vision backbone and predicts the temporal dynamics on local context patches incorporating meteorological time series in a parameter-efficient manner. The GreenEarthNet dataset features a learned cloud mask and an appropriate evaluation scheme for vegetation modeling. It also maintains compatibility with the existing satellite imagery forecasting dataset EarthNet2021, enabling cross-dataset model comparisons. Our extensive qualitative and quantitative analyses reveal that our methods outperform a broad range of baseline techniques. This includes surpassing previous state-of-the-art models on EarthNet2021, as well as adapted models from time series forecasting and video prediction. To the best of our knowledge, this work presents the first models for continental-scale vegetation modeling at fine resolution able to capture anomalies beyond the seasonal cycle, thereby paving the way for predicting vegetation health and behaviour in response to climate variability and extremes.

연구 동기 및 목표

• Extend EarthNet2021 for vegetation modeling with improved cloud masking and evaluation.
• Develop weather-conditioned, multi-model deep learning approaches for NDVI forecasting at 20m resolution across Europe.
• Compare meteo-guided models against strong baselines including non-ML methods and EarthNet2021 leaders.
• Demonstrate downstream applicability to gross primary productivity estimation for carbon monitoring.

제안 방법

• Predict future NDVI conditioned on past imagery, future weather, and elevation using deep neural networks.
• Evaluate four weather-guided model classes: ConvLSTM, PredRNN, SimVP, and Earthformer with meteo conditioning.
• Incorporate learned cloud masking and a new spatio-temporal OOD evaluation scheme to robustly assess vegetation prediction.
• Use a Gaussian likelihood objective with cloud and land-cover masks to train models on 20m NDVI targets.

Figure 1 : In this work, future vegetation status $\hat{V}$ is predicted with deep learning models $f$ from past satellite imagery $X$ , past and future weather $C$ and elevation $E$ . The underlying dataset spans across Europe with minicubes split into train (red dots), temporal OOD test (ood-t, or

실험 결과

연구 질문

• RQ1 Can continental-scale vegetation dynamics at 20m resolution be forecast from past Sentinel-2 imagery and weather data?
• RQ2 How does weather conditioning affect the skill of state-of-the-art video-prediction models in vegetation forecasting?
• RQ3 Which model class and weather-conditioning strategy best predicts NDVI across temporal and spatial out-of-distribution settings?
• RQ4 What is the downstream utility of predicted vegetation dynamics for carbon monitoring (gross primary production)?

주요 결과

모델	R^2↑	RMSE↓	NSE↑	|bias|↓	Outperform↑	RMSE 25 days↓	#매개변수
Persistence	0.00	0.23	-1.28	0.17	21.8%	0.09	0
Previous year	0.56	0.20	-0.40	0.14	19.3%	0.18	0
Climatology	0.58	0.18	-0.34	0.13	0.0%	0.16	0
Kalman filter	0.41	0.19	-0.57	0.13	27.0%	0.16	O(10)
LightGBM	0.51	0.17	-0.22	0.12	42.2%	0.11	n.a.
Prophet	0.57	0.16	-0.05	0.11	60.6%	0.13	O(10)
EN21 ConvLSTM	0.51	0.18	-0.37	0.12	43.9%	0.12	0.2M
SG-ConvLSTM	0.53	0.19	-0.33	0.14	45.8%	0.11	0.7M
Earthformer	0.49	0.17	-0.27	0.12	47.2%	0.11	60.6M
ConvLSTM-meteo	0.62 ± 0.01	0.14 ± 0.00	0.11 ± 0.03	0.10 ± 0.00	68.2% ± 1.8%	0.10 ± 0.00	1.0M
PredRNN-meteo	0.62 ± 0.00	0.15 ± 0.00	0.03 ± 0.00	0.10 ± 0.00	64.7% ± 1.2%	0.10 ± 0.00	1.4M
SimVP-meteo	0.60 ± 0.00	0.15 ± 0.00	0.03 ±0.01	0.09 ± 0.00	64.1% ± 1.0%	0.10 ± 0.00	6.6M
Earthformer-meteo	0.52	0.16	-0.13	0.10	56.5%	0.09	60.6M

• Weather-guided models outperform baselines across multiple metrics on Europe-wide 20m NDVI forecasting.
• ConvLSTM-meteo, PredRNN-meteo, and SimVP-meteo achieve higher skill than Earthformer-meteo and non-ML baselines in R^2, RMSE, NSE, and bias.
• Incorporating future weather with FiLM or cross-attention conditioning yields notable performance gains, with some methods benefiting from latent fusion strategies.
• The four models demonstrate robust predictive ability under temporal and spatial out-of-distribution tests (OOD-t and OOD-st).
• The NDVI predictions enable downstream estimation of gross primary productivity for carbon monitoring.

Figure 2 : Simplified view of evaluated models. Baselines (a,b), weather-guided deep learning (c,d,e,f).