[Paper Review] USat: A Unified Self-Supervised Encoder for Multi-Sensor Satellite Imagery

Jeremy Irvin, Lucas Tao+5 more|arXiv (Cornell University)|Dec 2, 2023

Remote-Sensing Image Classification4 citations

TL;DR

USat introduces a unified self-supervised vision transformer encoder for multi-sensor satellite imagery that handles spectral bands with varying ground sampling distances (GSD) without downscaling. By using sensor-specific patchification, superpositional positional encodings, and independent patch projection per band, USatMAE achieves state-of-the-art performance on remote sensing benchmarks, improving accuracy by up to 8% and showing strong gains in low-data regimes.

ABSTRACT

Large, self-supervised vision models have led to substantial advancements for automatically interpreting natural images. Recent works have begun tailoring these methods to remote sensing data which has rich structure with multi-sensor, multi-spectral, and temporal information providing massive amounts of self-labeled data that can be used for self-supervised pre-training. In this work, we develop a new encoder architecture called USat that can input multi-spectral data from multiple sensors for self-supervised pre-training. USat is a vision transformer with modified patch projection layers and positional encodings to model spectral bands with varying spatial scales from multiple sensors. We integrate USat into a Masked Autoencoder (MAE) self-supervised pre-training procedure and find that a pre-trained USat outperforms state-of-the-art self-supervised MAE models trained on remote sensing data on multiple remote sensing benchmark datasets (up to 8%) and leads to improvements in low data regimes (up to 7%). Code and pre-trained weights are available at https://github.com/stanfordmlgroup/USat .

Motivation & Objective

• To address the limitation of single-sensor self-supervised pre-training in remote sensing, which restricts model generalization and underutilizes complementary multi-sensor data.
• To develop a flexible encoder that can process spectral bands from multiple sensors with differing spatial resolutions (GSD) without downscaling, preserving spatial scale and information.
• To improve downstream performance on remote sensing tasks, especially in low-data regimes, by leveraging richer self-supervision from multi-sensor, multi-spectral data.
• To enable fine-tuning with arbitrary subsets of spectral bands and sensors, increasing model adaptability for diverse downstream applications.
• To overcome sequence length and memory inefficiencies in existing multi-sensor models by reducing sequence length through GSD-aware patchification and superpositional encodings.

Proposed method

• USat employs a vision transformer with sensor-specific patch projection layers that adaptively divide each spectral band into patches based on its ground sampling distance (GSD), ensuring higher-resolution bands contribute more patches.
• It applies superpositional positional encodings to model the hierarchical spatial structure of multi-GSD inputs, allowing higher-resolution bands to retain fine-grained spatial information.
• Patch embeddings from different spectral bands are combined via a spectral group pooling layer before being summed with positional encodings to form the final patch tokens for the transformer encoder.
• The model is pre-trained using a masked autoencoding (MAE) objective on multi-sensor data (e.g., Sentinel-2 and NAIP), reconstructing masked patches from the full input.
• The architecture supports arbitrary subsets of spectral bands and sensors during both pre-training and fine-tuning, enabling flexible downstream adaptation.
• The method avoids downscaling low-GSD (high-resolution) bands, preserving spatial detail lost in prior approaches that uniformly resize inputs.

Figure 1 : Overview of the USat encoder architecture. USat can accept any subset of spectral bands (channels) and spatial patches from multiple satellite (Sentinel-2) and aerial imagery (NAIP) sensors. Each spectral band is independently patchified, with lower GSD (higher spatial resolution) bands d

Experimental results

Research questions

• RQ1 Can a unified self-supervised encoder effectively integrate multi-sensor, multi-spectral satellite imagery with varying ground sampling distances (GSD) without downscaling?
• RQ2 Does jointly pre-training on multiple sensors (e.g., Sentinel-2 and NAIP) improve downstream performance compared to single-sensor pre-training?
• RQ3 Can a GSD-aware patchification and superpositional encoding scheme reduce sequence length and memory usage while preserving spatial information?
• RQ4 How does USatMAE perform in low-data regimes compared to supervised and single-sensor MAE baselines?
• RQ5 To what extent can the model be flexibly fine-tuned using arbitrary subsets of spectral bands from multiple sensors?

Key findings

• USatMAE outperforms state-of-the-art self-supervised MAE models on multiple remote sensing benchmarks, achieving up to 8% higher accuracy on average.
• On the USatlas dataset, USatMAE achieved 1.26 mAP improvement over supervised baselines and 0.41 mAP improvement over standard MAE.
• In low-data regimes, USatMAE improved performance by up to 7% compared to standard MAE, demonstrating strong few-shot generalization.
• The model achieved a 2.97 MAP improvement on METER-ML over supervised learning and a 3.16 MAP improvement over standard MAE.
• Joint pre-training on both Sentinel-2 and NAIP sensors outperformed pre-training on either sensor alone, confirming the benefit of multi-sensor self-supervision.
• By avoiding downscaling of high-resolution bands, USatMAE preserves spatial detail and avoids information loss inherent in prior multi-sensor approaches.

(a) EuroSAT