[Paper Review] A benchmark dataset for deep learning-based airplane detection: HRPlanes

Tolga Bakırman, Elif Sertel|arXiv (Cornell University)|Apr 22, 2022

Advanced Neural Network Applications39 references4 citations

TL;DR

This paper introduces HRPlanes, a novel high-resolution satellite imagery dataset specifically for airplane detection, curated from Google Earth images of major global airports and manually annotated with bounding boxes. Evaluated on YOLOv4 and Faster R-CNN, the dataset achieves over 90% mAP at 75% IoU, demonstrating its effectiveness as a benchmark for deep learning-based airplane detection and transfer learning applications.

ABSTRACT

Airplane detection from satellite imagery is a challenging task due to the complex backgrounds in the images and differences in data acquisition conditions caused by the sensor geometry and atmospheric effects. Deep learning methods provide reliable and accurate solutions for automatic detection of airplanes; however, huge amount of training data is required to obtain promising results. In this study, we create a novel airplane detection dataset called High Resolution Planes (HRPlanes) by using images from Google Earth (GE) and labeling the bounding box of each plane on the images. HRPlanes include GE images of several different airports across the world to represent a variety of landscape, seasonal and satellite geometry conditions obtained from different satellites. We evaluated our dataset with two widely used object detection methods namely YOLOv4 and Faster R-CNN. Our preliminary results show that the proposed dataset can be a valuable data source and benchmark data set for future applications. Moreover, proposed architectures and results of this study could be used for transfer learning of different datasets and models for airplane detection.

Motivation & Objective

• To address the lack of large-scale, diverse, and high-quality datasets specifically for airplane detection in satellite imagery.
• To create a benchmark dataset that captures variations in landscape, seasonality, and satellite acquisition geometry across global airports.
• To evaluate state-of-the-art object detection models (YOLOv4 and Faster R-CNN) on the proposed dataset to validate its utility.
• To enable transfer learning and model generalization across different remote sensing and airplane detection tasks.

Proposed method

• Collected high-resolution satellite images from Google Earth covering major international and military airports worldwide.
• Manually annotated bounding boxes for all visible airplanes in the images, ensuring high labeling accuracy.
• Constructed a balanced dataset with diverse conditions including varying weather, seasons, and sensor geometries.
• Trained and evaluated two leading object detection models—YOLOv4 and Faster R-CNN—on the dataset using standard metrics.
• Applied non-max suppression and IoU thresholding to assess detection performance and reduce false positives.
• Released model weights and test set via GitHub to support reproducibility and further research.

Experimental results

Research questions

• RQ1 Can a large-scale, diverse, and high-resolution satellite dataset improve the performance of deep learning models in airplane detection?
• RQ2 How do YOLOv4 and Faster R-CNN perform on a dataset with varied atmospheric and geometric conditions?
• RQ3 To what extent can the HRPlanes dataset serve as a benchmark for transfer learning in remote sensing applications?
• RQ4 What are the limitations of existing models in detecting partially visible or small-scale airplanes in complex backgrounds?

Key findings

• The HRPlanes dataset achieved over 90% mean average precision (mAP) at a 75% intersection over union (IoU) threshold on both YOLOv4 and Faster R-CNN.
• YOLOv4 demonstrated superior performance in detecting partially visible or small-scale airplanes, such as only nose cones or tail sections.
• Faster R-CNN showed higher sensitivity to motion blur and occlusion, occasionally missing parts of aircrafts even when YOLOv4 detected them.
• Both models successfully detected multiple airplanes in crowded scenes, thanks to effective non-max suppression.
• YOLOv4 generated more false positives in small-scale propeller aircraft scenes, but these could be reduced by increasing the confidence threshold to 0.50.
• The dataset proved effective for transfer learning, with model weights directly usable for inference on similar datasets or new remote sensing tasks.