[Paper Review] Extended Agriculture-Vision: An Extension of a Large Aerial Image Dataset for Agricultural Pattern Analysis

Jing Wu, David Pichler+4 more|arXiv (Cornell University)|Mar 4, 2023

Smart Agriculture and AI11 citations

TL;DR

The paper extends Agriculture-Vision with raw full-field imagery and large-scale unlabeled data for self-supervised pre-training, integrates Pixel-to-Propagation Module into MoCo-V2, and benchmarks CNN and Swin Transformer backbones on agricultural pattern analysis tasks.

ABSTRACT

A key challenge for much of the machine learning work on remote sensing and earth observation data is the difficulty in acquiring large amounts of accurately labeled data. This is particularly true for semantic segmentation tasks, which are much less common in the remote sensing domain because of the incredible difficulty in collecting precise, accurate, pixel-level annotations at scale. Recent efforts have addressed these challenges both through the creation of supervised datasets as well as the application of self-supervised methods. We continue these efforts on both fronts. First, we generate and release an improved version of the Agriculture-Vision dataset (Chiu et al., 2020b) to include raw, full-field imagery for greater experimental flexibility. Second, we extend this dataset with the release of 3600 large, high-resolution (10cm/pixel), full-field, red-green-blue and near-infrared images for pre-training. Third, we incorporate the Pixel-to-Propagation Module Xie et al. (2021b) originally built on the SimCLR framework into the framework of MoCo-V2 Chen et al.(2020b). Finally, we demonstrate the usefulness of this data by benchmarking different contrastive learning approaches on both downstream classification and semantic segmentation tasks. We explore both CNN and Swin Transformer Liu et al. (2021a) architectures within different frameworks based on MoCo-V2. Together, these approaches enable us to better detect key agricultural patterns of interest across a field from aerial imagery so that farmers may be alerted to problematic areas in a timely fashion to inform their management decisions. Furthermore, the release of these datasets will support numerous avenues of research for computer vision in remote sensing for agriculture.

Motivation & Objective

• Address the scarcity of large, accurately labeled remote sensing data for semantic segmentation in agriculture.
• Provide an extended, raw full-field dataset (AV+) for pre-training and evaluation.
• Benchmark self-supervised learning approaches on agricultural pattern analysis tasks using diverse backbones (CNN and Swin Transformer).
• Incorporate the Pixel-to-Propagation Module (PPM) into MoCo-V2 and adapt Temporal Contrast methods to AV+ to improve dense prediction tasks.

Proposed method

• Release full-field AV+ data containing raw RGB and NIR imagery for pre-training (3600 images, 10 cm/pixel GSD).
• Adapt MoCo-V2 for multi-channel (RGB+NIR) pre-training with instance-level contrast.
• Integrate Pixel-to-Propagation Module (PPM) for pixel-level pretext tasks and define PixPro loss for dense representations.
• Introduce temporal contrast (TemCo) to exploit multi-temporal AV+ data and adapt it with PPM (TemCo-PixPro).
• Explore Swin Transformer backbones (Swin-T) with MoCo-based pre-training and multi-head projections for temporal and pixel-level tasks.
• Use two downstream benchmarks: classification and semantic segmentation on AV+ with fixed and fine-tuned encoders.

Experimental results

Research questions

• RQ1 How does AV+ with raw full-field imagery and unlabeled data improve pre-training for agricultural pattern analysis?
• RQ2 What gains do MoCo-V2, MoCo-PixPro, TemCo, and TemCo-PixPro provide on downstream classification and segmentation tasks across CNN and Swin backbones?
• RQ3 Does incorporating PPM and multi-temporal contrast enhance dense predictions in aerial agricultural imagery?
• RQ4 How do RGB vs RGB+NIR channels affect performance in pre-training and downstream tasks?
• RQ5 What is the transferability of AV+-pretrained models to related remote sensing tasks (e.g., EuroSAT) and fine-grained segmentation within AV+?

Key findings

• Release of AV+ with 3600 raw full-field images for SSL pre-training (over 3 TB of unlabeled data).
• MoCo-PixPro and TemCo-PixPro consistently enhance downstream segmentation and classification over MoCo-V2 and ImageNet initializations, especially with smaller backbones and fixed encoders.
• Swim-T based MoCo variants show strong performance for segmentation when fully fine-tuned, outperforming ImageNet-initialized backbones in several settings.
• Pixel-level pretext tasks via PPM improve segmentation results, particularly as backbone capacity increases (ResNet-18 to Swin-T).
• Temporal contrast leveraging multi-time AV+ imagery (TemCo) and its combination with PPM (TemCo-PixPro) yield gains in temporal sensitivity for pattern analysis.
• Compared to Agriculture-Vision baselines, the Swin-T and SSL pre-training approaches achieve higher mean IoU in several configurations, especially with RGBN channels.