[Paper Review] MultiEarth 2023 -- Multimodal Learning for Earth and Environment Workshop and Challenge

Miriam Cha, Gregory Angelides+7 more|arXiv (Cornell University)|Jun 7, 2023

Seismology and Earthquake Studies4 citations

TL;DR

This paper presents MultiEarth 2023, a CVPR workshop and challenge focused on multimodal remote sensing for Earth and environmental monitoring, particularly in the Amazon rainforest. It introduces four sub-challenges—fire detection, deforestation estimation, environmental trend prediction, and SAR-to-optical image translation—using a curated dataset of aligned Sentinel-1, Sentinel-2, Landsat 5, and Landsat 8 imagery with deforestation and fire labels, enabling robust benchmarking of multimodal learning models for ecological monitoring under diverse conditions.

ABSTRACT

The Multimodal Learning for Earth and Environment Workshop (MultiEarth 2023) is the second annual CVPR workshop aimed at the monitoring and analysis of the health of Earth ecosystems by leveraging the vast amount of remote sensing data that is continuously being collected. The primary objective of this workshop is to bring together the Earth and environmental science communities as well as the multimodal representation learning communities to explore new ways of harnessing technological advancements in support of environmental monitoring. The MultiEarth Workshop also seeks to provide a common benchmark for processing multimodal remote sensing information by organizing public challenges focused on monitoring the Amazon rainforest. These challenges include estimating deforestation, detecting forest fires, translating synthetic aperture radar (SAR) images to the visible domain, and projecting environmental trends. This paper presents the challenge guidelines, datasets, and evaluation metrics. Our challenge website is available at https://sites.google.com/view/rainforest-challenge/multiearth-2023.

Motivation & Objective

• To advance multimodal representation learning for environmental monitoring by integrating diverse remote sensing data sources.
• To address critical ecological challenges in the Amazon rainforest, including deforestation and wildfires, through scalable machine learning methods.
• To establish a public benchmark for multimodal remote sensing by curating a continuous time-series dataset with aligned SAR, optical, and labeled deforestation and fire events.
• To enable robust evaluation of models across diverse environmental conditions using standardized metrics and challenge protocols.
• To foster collaboration between Earth scientists and machine learning researchers through a shared challenge framework and open data infrastructure.

Proposed method

• The challenge uses a multimodal dataset combining Sentinel-1 (SAR), Sentinel-2, Landsat 5, and Landsat 8 imagery, with temporal alignment within 7 days for SAR and optical pairs.
• For SAR-to-EO translation, models are trained to generate three possible RGB optical images (B4, B3, B2 bands) from a single SAR input (VV and VH bands), conditioned on spatial and temporal alignment.
• Evaluation for image translation uses a metric that computes the sum of minimum L2 distances between generated images and ground-truth EO images: $\sum_{j} \min_{i} \| f(\mathbf{x})_i - \mathbf{y}_j \| $, promoting fidelity and diversity.
• Participants are required to generate 15,000 output images (3 per SAR input) for 5,000 test samples, with specific file naming and data format standards (NetCDF or TIFF with uint16 dtype).
• The dataset includes 5,000 SAR images (2,500 VV and 2,500 VH) with corresponding geographic coordinates and timestamps, enabling time-series modeling.
• A public API and tutorial are provided to assist researchers in data retrieval, filtering, and processing, ensuring reproducibility and ease of access.

Experimental results

Research questions

• RQ1 How effectively can multimodal deep learning models detect forest fires using fused SAR and optical remote sensing data?
• RQ2 To what extent can models accurately estimate deforestation from multimodal remote sensing inputs under varying weather and lighting conditions?
• RQ3 Can generative models learn to synthesize realistic, diverse, and temporally consistent optical images from SAR inputs across the Amazon rainforest?
• RQ4 How well can models predict future environmental states of the Amazon rainforest using historical multimodal time-series data?
• RQ5 What is the performance of state-of-the-art multimodal representation learning models in handling the unique challenges of Earth observation data, such as sensor heterogeneity and temporal misalignment?

Key findings

• The MultiEarth 2023 challenge provides a comprehensive benchmark dataset with aligned multimodal remote sensing data from Sentinel-1, Sentinel-2, Landsat 5, and Landsat 8, covering the Amazon rainforest with continuous temporal coverage.
• The dataset includes 5,000 test samples with paired SAR and optical images, where each SAR image is associated with an average of three EO images collected within 7 days.
• The SAR-to-EO image translation sub-challenge requires generating three distinct, high-fidelity optical images per SAR input, with evaluation based on both accuracy and diversity using a minimum-distance metric.
• The challenge supports two data formats—NetCDF and TIFF—with standardized naming conventions and data types (uint16) to ensure consistent model evaluation.
• The dataset includes ground-truth labels for deforestation and forest fires, enabling performance evaluation on critical ecological tasks using metrics such as F1 score, IoU, and pixel accuracy.
• A public API and tutorial are provided to facilitate data access and processing, lowering the barrier to entry for researchers in Earth observation and machine learning.