[Paper Review] CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoders

Anthony T. Fuller, Koreen Millard+1 more|arXiv (Cornell University)|Nov 1, 2023

Synthetic Aperture Radar (SAR) Applications and Techniques14 citations

TL;DR

CROMA combines contrastive radar–optical learning with masked autoencoding to learn rich unimodal and multimodal remote sensing representations, enabling extrapolation to larger images and outperforming prior multispectral models on several benchmarks.

ABSTRACT

A vital and rapidly growing application, remote sensing offers vast yet sparsely labeled, spatially aligned multimodal data; this makes self-supervised learning algorithms invaluable. We present CROMA: a framework that combines contrastive and reconstruction self-supervised objectives to learn rich unimodal and multimodal representations. Our method separately encodes masked-out multispectral optical and synthetic aperture radar samples -- aligned in space and time -- and performs cross-modal contrastive learning. Another encoder fuses these sensors, producing joint multimodal encodings that are used to predict the masked patches via a lightweight decoder. We show that these objectives are complementary when leveraged on spatially aligned multimodal data. We also introduce X- and 2D-ALiBi, which spatially biases our cross- and self-attention matrices. These strategies improve representations and allow our models to effectively extrapolate to images up to 17.6x larger at test-time. CROMA outperforms the current SoTA multispectral model, evaluated on: four classification benchmarks -- finetuning (avg. 1.8%), linear (avg. 2.4%) and nonlinear (avg. 1.4%) probing, kNN classification (avg. 3.5%), and K-means clustering (avg. 8.4%); and three segmentation benchmarks (avg. 6.4%). CROMA's rich, optionally multimodal representations can be widely leveraged across remote sensing applications.

Motivation & Objective

• Motivate and address the scarcity of labeled data in remote sensing by learning rich self-supervised representations from spatially aligned multimodal data (Sentinel-1 SAR and Sentinel-2 optical).
• Develop a framework that combines contrastive learning and masked autoencoding to learn unimodal and multimodal representations.
• Introduce spatial biasing in attention (2D-ALiBi and X-ALiBi) to improve generalization and cross-modal fusion, enabling test-time extrapolation to larger image sizes.

Proposed method

• Three encoders process radar, optical, and joint radar–optical inputs (ViT-based).
• Masked autoencoding objective reconstructs masked patches from both modalities using a lightweight decoder.
• Radar↔optical contrastive loss aligns unimodal representations across modalities.
• Cross-modal multimodal encoder fRO learns joint representations via cross-attention to optical encodings.
• 2D-ALiBi biases self-attention for 2D patch distances, and X-ALiBi biases cross-attention to improve fusion.
• A multimodal reconstruction target (14 channels) enhances multimodal representation learning beyond optical-only targets.

Experimental results

Research questions

• RQ1 Can a joint radar–optical self-supervised framework outperform single-modality pretraining in remote sensing tasks?
• RQ2 Do reconstruction and contrastive objectives complement each other when learning from spatially aligned multimodal remote sensing data?
• RQ3 How do 2D-ALiBi and X-ALiBi influence extrapolation to larger image sizes and cross-modal fusion?

Key findings

• CROMA outperforms the current state-of-the-art multispectral model SatMAE across four classification benchmarks when evaluated with finetuning, linear, and nonlinear probing, as well as kNN and K-means clustering.
• CROMA achieves stronger segmentation performance on three Sentinel-2 benchmarks, surpassing SatMAE on average for both ViT-B and ViT-L backbones.
• Joint multimodal representations (radar–optical) improve performance over optical-only representations, with notable gains in BigEarthNet and Dynamic World benchmarks.
• CROMA demonstrates the ability to extrapolate to images up to 17.6× larger at test time with modest degradation, thanks to 2D-ALiBi and X-ALiBi biases.
• Compared to radar-only and radar–optical baselines, CROMA’s multimodal representations show strong linear probing performance and competitiveness against concurrent multimodal methods like DeCUR.
• Ablation studies confirm that combining contrastive and reconstruction objectives and the proposed positional biases are key to performance and extrapolation capabilities.