[Paper Review] SeaMo: A Season-Aware Multimodal Foundation Model for Remote Sensing

Xuyang Li, Chen-Yu Li+2 more|arXiv (Cornell University)|Dec 26, 2024

Advanced Computational Techniques and Applications4 citations

TL;DR

SeaMo is a novel season-aware, multimodal foundation model for remote sensing that integrates multi-seasonal and multi-source data (optical, SAR) via a progressive masked image modeling framework. It leverages non-aligned cropping, modality-specific tokenizers, and temporal-multimodal fusion blocks to learn robust, invariant representations, achieving state-of-the-art performance across diverse downstream tasks like classification, change detection, and segmentation on datasets including EuroSAT, fMoW, and BigEarthNet.

ABSTRACT

Remote Sensing (RS) data encapsulates rich multi-dimensional information essential for Earth observation. Its vast volume, diverse sources, and temporal continuity make it particularly well-suited for developing large Visual Foundation Models (VFMs). These models serve as powerful feature extractors, leveraging extensive RS data for pretraining and subsequent fine-tuning in various geoscientific applications. However, existing VFMs in the RS domain often concentrate on specific image characteristics, neglecting the full season-aware potential of RS data. To bridge this gap, we introduce SeaMo, a novel VFM that effectively integrates multimodal and multi-seasonal RS information. SeaMo leverages a masked image modeling framework to fully exploit the spatial, spectral, and seasonal dimensions of RS data. Specifically, we employ unaligned spatial region selection to capture spatial heterogeneity, incorporate multi-source inputs for enhanced multimodal integration, and introduce temporal-multimodal fusion blocks to assimilate seasonal variations effectively. By explicitly modeling the complex, season-dependent attributes of RS data, SeaMo enhances generalization, robustness, and adaptability across geoscientific tasks. Extensive experiments and ablation studies demonstrate its superior performance, underscoring its potential as a foundational model for Earth observation.

Motivation & Objective

• To address the limitation of existing remote sensing foundation models that focus on single-dimensional properties (e.g., spatial or temporal) and neglect the integration of multi-seasonal and multimodal data.
• To develop a self-supervised visual foundation model that explicitly models the multi-dimensional nature of remote sensing data, including spatial, temporal, and spectral characteristics.
• To improve feature extraction and downstream task performance by incorporating heterogeneous satellite imagery through a progressive pretraining strategy.
• To validate the model's robustness and generalizability across varying image sizes and spectral band counts in real-world remote sensing applications.

Proposed method

• SeaMo employs a masked image modeling (MIM) framework with progressive pretraining, gradually increasing data complexity to refine representation learning.
• Non-aligned cropping is used to extract spatial properties from heterogeneous satellite images, enhancing spatial invariance and robustness.
• Multi-source inputs (e.g., optical and SAR) are processed through modality-specific tokenizers and fused via temporal-multimodal fusion blocks (TM blocks) to capture cross-modal dependencies.
• A positional encoding interpolation strategy enables generalization to varying input image sizes, while a new tokenizer is introduced for different spectral band counts.
• The model uses a shared vision transformer backbone with a learnable mask token and a reconstruction head to predict masked patches in a self-supervised manner.
• The pretraining pipeline is designed to progressively incorporate more complex data, including multi-seasonal sequences and multimodal inputs, to improve feature generalization.

Experimental results

Research questions

• RQ1 How can a foundation model effectively integrate multi-seasonal and multimodal remote sensing data to improve representation learning?
• RQ2 What architectural components are essential for capturing the multi-dimensional properties (spatial, temporal, spectral, modal) of remote sensing data?
• RQ3 To what extent does progressive self-supervised pretraining enhance the robustness and generalization of foundation models in remote sensing?
• RQ4 How does the model perform across diverse downstream tasks, including classification, segmentation, and change detection, on heterogeneous data sources?
• RQ5 What is the sensitivity of the model to variations in input image size and spectral band count in real-world deployment scenarios?

Key findings

• SeaMo achieves a test accuracy of 99.37% on the EuroSAT dataset when fine-tuned on 128×128×12 optical and SAR data, demonstrating strong generalization and robustness.
• Performance improves with longer pretraining durations, with ablation studies showing diminishing returns beyond a certain point, indicating effective convergence.
• The model maintains high performance (99.26% accuracy) on 224×224 images, indicating minimal sensitivity to input size, but performance drops significantly (to 99.08%) when spectral bands are reduced to 4.
• The temporal-multimodal fusion blocks (TM blocks) are critical for integrating multi-seasonal and multimodal data, as shown by ablation studies that degrade performance when removed.
• The model outperforms existing foundation models across multiple downstream tasks, including change detection on OSCD and semantic segmentation on DFC2020, confirming its versatility and effectiveness.
• Abllation studies confirm that non-aligned cropping and progressive pretraining are key to learning spatial invariance and robust temporal representations, respectively.