[Paper Review] RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing

Zilun Zhang, Tiancheng Zhao+2 more|arXiv (Cornell University)|Jun 20, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper introduces RS5M, a large-scale remote sensing (RS) image-text dataset with 5 million paired images and descriptions, and GeoRSCLIP, a domain-specific vision-language model fine-tuned on RS5M. Using parameter-efficient fine-tuning on CLIP, GeoRSCLIP achieves state-of-the-art performance, improving zero-shot classification by 3–20%, retrieval by 3–6%, and semantic localization by 4–5% over baselines.

ABSTRACT

Pre-trained Vision-Language Models (VLMs) utilizing extensive image-text paired data have demonstrated unprecedented image-text association capabilities, achieving remarkable results across various downstream tasks. A critical challenge is how to make use of existing large-scale pre-trained VLMs, which are trained on common objects, to perform the domain-specific transfer for accomplishing domain-related downstream tasks. A critical challenge is how to make use of existing large-scale pre-trained VLMs, which are trained on common objects, to perform the domain-specific transfer for accomplishing domain-related downstream tasks. In this paper, we propose a new framework that includes the Domain pre-trained Vision-Language Model (DVLM), bridging the gap between the General Vision-Language Model (GVLM) and domain-specific downstream tasks. Moreover, we present an image-text paired dataset in the field of remote sensing (RS), RS5M, which has 5 million RS images with English descriptions. The dataset is obtained from filtering publicly available image-text paired datasets and captioning label-only RS datasets with pre-trained VLM. These constitute the first large-scale RS image-text paired dataset. Additionally, we fine-tuned the CLIP model and tried several Parameter-Efficient Fine-Tuning methods on RS5M to implement the DVLM. Experimental results show that our proposed dataset is highly effective for various tasks, and our model GeoRSCLIP improves upon the baseline or previous state-of-the-art model by $3\%\sim20\%$ in Zero-shot Classification (ZSC), $3\%\sim6\%$ in Remote Sensing Cross-Modal Text-Image Retrieval (RSCTIR) and $4\%\sim5\%$ in Semantic Localization (SeLo) tasks. Dataset and models have been released in: \url{https://github.com/om-ai-lab/RS5M}.

Motivation & Objective

• To address the lack of large-scale, high-quality remote sensing image-text paired datasets for training domain-specific vision-language models.
• To bridge the domain gap between general vision-language models (GVLMs) and remote sensing applications, where domain-specific knowledge is critical.
• To develop a Domain-specific Vision-Language Model (DVLM) that effectively transfers generalizable knowledge from GVLMs to RS-specific downstream tasks.
• To evaluate the effectiveness of parameter-efficient fine-tuning methods on large-scale RS data for improved model generalization.
• To release RS5M and GeoRSCLIP as open resources to accelerate research in remote sensing vision-language understanding.

Proposed method

• Constructed RS5M by filtering and augmenting publicly available image-text datasets using a pre-trained vision-language model to ensure high-quality, domain-relevant image-text pairs.
• Leveraged existing large-scale datasets such as LAION2B, COYO700M, WIT, and YFCC15M, applying strict filtering to retain only remote sensing images with descriptive captions.
• Fine-tuned the CLIP backbone using parameter-efficient fine-tuning (PEFT) techniques, including LoRA, to adapt the general vision-language model to the remote sensing domain.
• Evaluated the resulting GeoRSCLIP model on three downstream tasks: zero-shot classification, cross-modal retrieval, and semantic localization.
• Generated a specialized Stable Diffusion model (RS-SD) by fine-tuning on 1% of RS5M data using Dreambooth, improving image generation fidelity for remote sensing.
• Performed extensive ablation studies and safety checks, including metadata license validation and removal of sensitive geolocation data, to ensure ethical and legal compliance.

Experimental results

Research questions

• RQ1 Can a large-scale, high-quality image-text dataset significantly improve the performance of vision-language models in remote sensing applications?
• RQ2 How effective is parameter-efficient fine-tuning in adapting a general vision-language model to the remote sensing domain?
• RQ3 To what extent does domain-specific pre-training on RS5M improve zero-shot classification, retrieval, and localization performance compared to general models?
• RQ4 Can a vision-language model fine-tuned on RS5M generate more realistic and semantically aligned remote sensing images than a vanilla diffusion model?
• RQ5 What are the key challenges and risks in constructing a large-scale RS image-text dataset, and how can they be mitigated?

Key findings

• GeoRSCLIP achieves a 3% to 20% improvement in zero-shot classification accuracy over the baseline CLIP model on remote sensing benchmarks.
• The model improves retrieval performance by 3% to 6% in Remote Sensing Cross-Modal Text–Image Retrieval (RSCTIR) tasks.
• Semantic localization (SeLo) performance improves by 4% to 5% compared to the baseline and previous state-of-the-art models.
• The fine-tuned Stable Diffusion model (RS-SD), trained on 1% of RS5M, achieves a Fréchet Inception Distance (FID) score of 28.32, significantly better than the vanilla Stable Diffusion’s 36.86, indicating superior image generation quality.
• Outlier analysis shows only 0.8% of RS5M images are misclassified as non-RS, and 3.4% of removed images were incorrectly filtered, indicating high data quality and robust filtering.
• The dataset and models are released under a non-commercial academic use license, with full metadata and image-text pairs available via GitHub and cloud storage for research use.