[Paper Review] Geographical Knowledge-driven Representation Learning for Remote Sensing Images

Wenyuan Li, Keyan Chen+2 more|arXiv (Cornell University)|Jul 12, 2021

Remote-Sensing Image ClassificationEngineering73 references73 citations

TL;DR

This paper proposes GeoKR, a geographical knowledge-driven representation learning method for remote sensing images that leverages GlobeLand30 land cover data and geographical locations as weak supervision to pre-train deep networks. By using a mean teacher framework to mitigate noise from resolution and timing mismatches, GeoKR achieves state-of-the-art performance on downstream tasks like scene classification, semantic segmentation, and cloud/snow detection, significantly reducing annotation needs—outperforming ImageNet pre-training and self-supervised methods with up to 50% less labeled data.

ABSTRACT

The proliferation of remote sensing satellites has resulted in a massive amount of remote sensing images. However, due to human and material resource constraints, the vast majority of remote sensing images remain unlabeled. As a result, it cannot be applied to currently available deep learning methods. To fully utilize the remaining unlabeled images, we propose a Geographical Knowledge-driven Representation learning method for remote sensing images (GeoKR), improving network performance and reduce the demand for annotated data. The global land cover products and geographical location associated with each remote sensing image are regarded as geographical knowledge to provide supervision for representation learning and network pre-training. An efficient pre-training framework is proposed to eliminate the supervision noises caused by imaging times and resolutions difference between remote sensing images and geographical knowledge. A large scale pre-training dataset Levir-KR is proposed to support network pre-training. It contains 1,431,950 remote sensing images from Gaofen series satellites with various resolutions. Experimental results demonstrate that our proposed method outperforms ImageNet pre-training and self-supervised representation learning methods and significantly reduces the burden of data annotation on downstream tasks such as scene classification, semantic segmentation, object detection, and cloud / snow detection. It demonstrates that our proposed method can be used as a novel paradigm for pre-training neural networks. Codes will be available on https://github.com/flyakon/Geographical-Knowledge-driven-Representaion-Learning.

Motivation & Objective

• To address the scarcity of labeled remote sensing data by leveraging millions of unlabeled images.
• To improve deep learning performance on downstream tasks such as scene classification, semantic segmentation, object detection, and cloud/snow detection.
• To reduce the annotation burden by using geographical knowledge as weak supervision for pre-training.
• To develop a robust pre-training framework that mitigates noise from resolution and imaging time discrepancies between remote sensing images and geographical knowledge.

Proposed method

• Uses GlobeLand30 global land cover product and geographical coordinates as geographical knowledge to generate supervision signals.
• Constructs knowledge representations by mapping land cover types to embedding vectors for supervision.
• Employs a dual-network architecture (student and teacher) with weight averaging to stabilize training and reduce noise from image-knowledge mismatches.
• Applies KL divergence loss to align image representations with knowledge representations during pre-training.
• Introduces a contrastive learning strategy via the teacher network to improve feature generalization and robustness.
• Builds the Levir-KR dataset with 1,431,950 Gaofen satellite images across multiple resolutions for pre-training.

Experimental results

Research questions

• RQ1 Can geographical knowledge such as land cover and location be effectively used to supervise representation learning in remote sensing images?
• RQ2 How can noise from resolution and imaging time differences between remote sensing images and geographical knowledge be mitigated during pre-training?
• RQ3 To what extent can GeoKR reduce the need for annotated data in downstream remote sensing tasks?
• RQ4 Does GeoKR outperform ImageNet pre-training and self-supervised learning methods in downstream performance?

Key findings

• GeoKR outperforms ImageNet pre-training and self-supervised methods (MoCo, SimCLR, BYOL) across all downstream tasks, including scene classification, semantic segmentation, object detection, and cloud/snow detection.
• On the RSSCN7 dataset, GeoKR (ResNet50) achieves 94.19% accuracy with 100% training data, surpassing the baseline by over 2%.
• For semantic segmentation on Vaihingen, GeoKR (ResNet50) reaches 72.68% accuracy with 100% data, showing a 2.5% improvement over the baseline.
• With only 20% of the training data, GeoKR achieves performance comparable to ImageNet pre-training using 100% data in scene classification, reducing annotation needs by half.
• In snow detection, GeoKR improves performance by over 10% when using limited data, demonstrating strong data efficiency.
• Abllation studies confirm that the teacher network and knowledge representation are critical components, with the teacher model yielding the best results.