[Paper Review] GenCo: An Auxiliary Generator from Contrastive Learning for Enhanced Few-Shot Learning in Remote Sensing

Jing Wu, Naira Hovakimyan+1 more|arXiv (Cornell University)|Jul 27, 2023

Domain Adaptation and Few-Shot LearningComputer Science3 citations

TL;DR

This paper proposes GenCo, a contrastive learning framework with an auxiliary generator that enhances few-shot learning in remote sensing by pre-training on unlabeled aerial imagery and generating synthetic feature samples during fine-tuning. It achieves state-of-the-art performance on Agriculture-Vision and EuroSAT, outperforming ImageNet pre-training even with only 10 labeled samples per class.

ABSTRACT

Classifying and segmenting patterns from a limited number of examples is a significant challenge in remote sensing and earth observation due to the difficulty in acquiring accurately labeled data in large quantities. Previous studies have shown that meta-learning, which involves episodic training on query and support sets, is a promising approach. However, there has been little attention paid to direct fine-tuning techniques. This paper repurposes contrastive learning as a pre-training method for few-shot learning for classification and semantic segmentation tasks. Specifically, we introduce a generator-based contrastive learning framework (GenCo) that pre-trains backbones and simultaneously explores variants of feature samples. In fine-tuning, the auxiliary generator can be used to enrich limited labeled data samples in feature space. We demonstrate the effectiveness of our method in improving few-shot learning performance on two key remote sensing datasets: Agriculture-Vision and EuroSAT. Empirically, our approach outperforms purely supervised training on the nearly 95,000 images in Agriculture-Vision for both classification and semantic segmentation tasks. Similarly, the proposed few-shot method achieves better results on the land-cover classification task on EuroSAT compared to the results obtained from fully supervised model training on the dataset.

Motivation & Objective

• Address the challenge of data scarcity in remote sensing by improving few-shot learning performance for classification and semantic segmentation tasks.
• Overcome the limitations of existing contrastive learning methods, which are primarily optimized for natural scene images and do not generalize well to remote sensing data.
• Develop a pre-training strategy that leverages abundant unlabeled remote sensing data to learn robust, transferable representations for downstream few-shot tasks.
• Introduce an auxiliary generator during pre-training that synthesizes diverse feature variants, enriching the representation learning process and improving generalization under label scarcity.
• Demonstrate that pre-training on domain-specific remote sensing data (e.g., Agriculture-Vision) yields superior few-shot performance compared to ImageNet or COCO pre-training.

Proposed method

• Propose a two-stage training framework: contrastive pre-training followed by few-shot fine-tuning, with the encoder weights frozen during fine-tuning.
• Introduce an auxiliary generator network that learns to produce diverse, positive feature variants from encoded features during contrastive pre-training.
• Use the generator to create additional positive pairs in feature space, increasing the contrastive signal and improving representation robustness.
• Pre-train the backbone and generator on the Extended Agriculture-Vision (AV+) dataset using contrastive learning objectives such as InfoNCE, with data augmentations applied to input images.
• During fine-tuning, utilize the pre-trained generator to generate synthetic feature samples in latent space, enriching the limited labeled data and improving generalization.
• Fine-tune only the classification head or segmentation decoder while keeping the encoder and generator frozen, enabling efficient adaptation to few-shot scenarios.

Experimental results

Research questions

• RQ1 Can a generator-based contrastive learning framework improve few-shot classification and segmentation performance on remote sensing datasets compared to standard pre-training methods?
• RQ2 Does pre-training on domain-specific remote sensing data (e.g., AV+) yield better few-shot generalization than pre-training on general-image datasets like ImageNet or COCO?
• RQ3 To what extent can an auxiliary generator that synthesizes feature variants during pre-training enhance representation learning and downstream few-shot performance?
• RQ4 How does the proposed method compare to fully supervised training when only a small number of labeled samples are available?
• RQ5 Does the performance gain from GenCo persist across different backbone architectures and few-shot shot settings (1-shot, 5-shot, 10-shot) on diverse remote sensing benchmarks?

Key findings

• GenCo outperforms ImageNet pre-training by 1 to 6 percentage points on the 10-shot classification task across multiple backbones on the Agriculture-Vision dataset.
• On the Agriculture-Vision semantic segmentation task, GenCo improves mIoU by 5 to 7 points compared to COCO-pretrained features under the same few-shot setting.
• On EuroSAT, GenCo achieves 67.92% accuracy with ResNet-18 on the 10-shot classification task, surpassing ImageNet pre-training by 1.02 percentage points and outperforming MoCo-V2, SimSiam, and SimCLR pre-trained on AV+.
• With only 10 labeled images per class, GenCo achieves 66.90% accuracy on EuroSAT, outperforming a fully supervised model trained on 27,000 images by 3.66 percentage points.
• The method maintains consistent gains across different backbone sizes (ResNet-18, -50, -101), with minimal performance drop in 1-shot and 5-shot settings, indicating strong generalization under extreme label scarcity.
• The auxiliary generator significantly enhances representation quality, as evidenced by improved performance even when fine-tuning with minimal labeled data, proving its value in data-scarce remote sensing applications.