[Paper Review] Multi-Region Transfer Learning for Segmentation of Crop Field Boundaries in Satellite Images with Limited Labels

Hannah Kerner, Saketh Sundar+1 more|arXiv (Cornell University)|Mar 29, 2024

Remote Sensing in Agriculture6 citations

TL;DR

The paper introduces a multi-region transfer learning approach to segment crop field boundaries in satellite images when target-region labels are scarce, using a three-stage pre-train/fine-tune/evaluate pipeline with a Spatio-Temporal U-net and multi-temporal inputs.

ABSTRACT

The goal of field boundary delineation is to predict the polygonal boundaries and interiors of individual crop fields in overhead remotely sensed images (e.g., from satellites or drones). Automatic delineation of field boundaries is a necessary task for many real-world use cases in agriculture, such as estimating cultivated area in a region or predicting end-of-season yield in a field. Field boundary delineation can be framed as an instance segmentation problem, but presents unique research challenges compared to traditional computer vision datasets used for instance segmentation. The practical applicability of previous work is also limited by the assumption that a sufficiently-large labeled dataset is available where field boundary delineation models will be applied, which is not the reality for most regions (especially under-resourced regions such as Sub-Saharan Africa). We present an approach for segmentation of crop field boundaries in satellite images in regions lacking labeled data that uses multi-region transfer learning to adapt model weights for the target region. We show that our approach outperforms existing methods and that multi-region transfer learning substantially boosts performance for multiple model architectures. Our implementation and datasets are publicly available to enable use of the approach by end-users and serve as a benchmark for future work.

Motivation & Objective

• Motivate field boundary delineation as an important, practical segmentation task for agriculture (yield, area estimates, surveys).
• Address label scarcity in target regions by leveraging labeled data from other regions through multi-region transfer learning.
• Develop and evaluate a flexible architecture that works across regions and sensors to improve boundary and interior delineation.

Proposed method

• Adopt the Spatio-Temporal U-net (ST-U-net) architecture with a ResNet backbone to predict border and interior masks.
• Use multi-region transfer learning: pre-train on a large labeled dataset from region Dp, fine-tune on a smaller region-f dataset, and evaluate on the target region Dt, with the fine-tuning region acting as a bridge between Dp and Dt.
• Incorporate multi-temporal inputs (three seasonal composites) to capture temporal evolution of fields, comparing against a single-time-input U-net.
• Process data to 224x224 image tiles across Sentinel-2 and PlanetScope datasets, with three seasonal composites per region.
• Evaluate both border (boundary) and interior masks, using pixel-wise F1, accuracy, mean IoU, and precision at IoU >= 0.95 as metrics.

Figure 1: Comparison between instance segmentation of crop fields in satellite remote sensing images (top) and example image from COCO dataset (bottom). In the top row, $t_{0}$ , $t_{1}$ , and $t_{n}$ represent satellite images acquired at different dates during the year. Top image shows field bound

Experimental results

Research questions

• RQ1 Can multi-region transfer learning improve field boundary segmentation in regions with limited labeled data?
• RQ2 Does multi-temporal input (three seasonal composites) outperform single-time inputs for boundary and interior delineation?
• RQ3 How do cross-region and cross-sensor transfers (e.g., Sentinel-2 to PlanetScope) affect performance across different regions?
• RQ4 What is the impact of fine-tuning with a bridging region on target-region performance?
• RQ5 Is the proposed approach robust across diverse regions (France, South Africa, Kenya) and label availability scenarios?

Key findings

• ST-U-net consistently outperforms a U-net with the same backbone across metrics and regions.
• Using three temporal inputs significantly improves performance over single-time inputs.
• Deeper backbones (ResNet-50, ResNet-101) yield higher performance but require more training time.
• Fine-tuning with bridging-region data (e.g., France S2 to South Africa PS) substantially improves border and interior metrics in Kenya and other test sets.
• Cross-region transfer (pre-training in one region, fine-tuning in another, testing in a third) demonstrates notable gains over no-fine-tune baselines.
• Unsupervised baselines (Canny + watershed) underperform compared to learned models.

Figure 2: Illustration of multi-region transfer learning approach. Model architecture adapted from (Aung et al. 2020 ) .