[Paper Review] iSAID: A Large-scale Dataset for Instance Segmentation in Aerial Images

Syed Waqas Zamir, Aditya Arora+8 more|arXiv (Cornell University)|May 30, 2019

Advanced Neural Network Applications33 references161 citations

TL;DR

iSAID introduces a large-scale, densely annotated dataset for instance segmentation in aerial imagery with 15 categories and 655,451 instances across 2,806 high-resolution images, and benchmarks common methods (Mask R-CNN, PANet) revealing the need for aerial-specific solutions.

ABSTRACT

Existing Earth Vision datasets are either suitable for semantic segmentation or object detection. In this work, we introduce the first benchmark dataset for instance segmentation in aerial imagery that combines instance-level object detection and pixel-level segmentation tasks. In comparison to instance segmentation in natural scenes, aerial images present unique challenges e.g., a huge number of instances per image, large object-scale variations and abundant tiny objects. Our large-scale and densely annotated Instance Segmentation in Aerial Images Dataset (iSAID) comes with 655,451 object instances for 15 categories across 2,806 high-resolution images. Such precise per-pixel annotations for each instance ensure accurate localization that is essential for detailed scene analysis. Compared to existing small-scale aerial image based instance segmentation datasets, iSAID contains 15$ imes$ the number of object categories and 5$ imes$ the number of instances. We benchmark our dataset using two popular instance segmentation approaches for natural images, namely Mask R-CNN and PANet. In our experiments we show that direct application of off-the-shelf Mask R-CNN and PANet on aerial images provide suboptimal instance segmentation results, thus requiring specialized solutions from the research community. The dataset is publicly available at: https://captain-whu.github.io/iSAID/index.html

Motivation & Objective

• Motivate the creation of a large-scale, densely annotated instance segmentation dataset for aerial images.
• Provide precise per-pixel instance masks to enable accurate localization in high-density aerial scenes.
• Highlight the limitations of applying natural-scene instance segmentation models to aerial imagery.
• Benchmark baseline methods to establish a reference and motivate developing aerial-tailored approaches.

Proposed method

• Curate and annotate 2,806 high-resolution aerial images with 655,451 instance masks across 15 categories using a dedicated annotation pipeline.
• Establish rigorous annotation guidelines, a multi-stage quality-control workflow, and professional annotators to ensure per-instance pixel-level masks.
• Evaluate baseline instance segmentation and object detection methods (Mask R-CNN and PANet) on iSAID, including simple architectural/backbone enhancements and scale augmentation.
• Adapt evaluation for high-resolution aerial imagery by using 800x800 patches with stride 200 and re-annotating cropped patches as needed.
• Provide benchmark results for both instance segmentation and bounding-box detection to reveal performance gaps on aerial data.

Experimental results

Research questions

• RQ1 How large and diverse must an aerial-instance segmentation dataset be to train robust models?
• RQ2 What is the performance gap between state-of-the-art natural-image methods and aerial-image data for instance segmentation?
• RQ3 How do common architectures (Mask R-CNN, PANet) fare on dense, high-resolution aerial imagery, and can simple aerial-specific tweaks close the gap?
• RQ4 What are the challenges (scale, density, orientation, aspect ratio) that affect instance segmentation in aerial images?

Key findings

• iSAID contains 655,451 annotated instances across 2,806 images and 15 categories, enabling high-density aerial scene analysis.
• Off-the-shelf natural-image methods (Mask R-CNN, PANet) yield suboptimal results on iSAID, indicating the need for aerial-tailored solutions.
• PANet variants outperform Mask R-CNN variants on iSAID, with PANet++ (ResNet-152-FPN backbone) achieving the best performance in instance segmentation and bounding-box detection.
• Simple modifications to baselines (scale augmentation, higher box counts, adjusted NMS) provide meaningful improvements, illustrating the potential gains from domain-specific tweaks.
• Class-wise results show notable improvements for certain categories (e.g., baseball diamond, basketball court, harbor) with PANet++.
• Compared to DOTA, iSAID introduces more instances and categories, presenting a more challenging benchmark for aerial instance segmentation.