[Paper Review] Rethinking Drone-Based Search and Rescue with Aerial Person Detection

Pasi Pyrrö, Hassan Naseri+1 more|arXiv (Cornell University)|Nov 17, 2021

Advanced Neural Network Applications41 references4 citations

TL;DR

This paper proposes the Aerial Inspection RetinaNet (AIR), a deep learning model that automates aerial person detection in drone-based search and rescue (SAR) missions. By integrating a novel postprocessing method—Merging of Overlapping Bounding boxes (MOB)—and a new SAR-specific evaluation scheme, AIR achieves a 21-percentage-point increase in precision (94.9%) and 92.9% recall on the HERIDAL benchmark, significantly outperforming human inspectors and state-of-the-art methods.

ABSTRACT

The visual inspection of aerial drone footage is an integral part of land search and rescue (SAR) operations today. Since this inspection is a slow, tedious and error-prone job for humans, we propose a novel deep learning algorithm to automate this aerial person detection (APD) task. We experiment with model architecture selection, online data augmentation, transfer learning, image tiling and several other techniques to improve the test performance of our method. We present the novel Aerial Inspection RetinaNet (AIR) algorithm as the combination of these contributions. The AIR detector demonstrates state-of-the-art performance on a commonly used SAR test data set in terms of both precision (~21 percentage point increase) and speed. In addition, we provide a new formal definition for the APD problem in SAR missions. That is, we propose a novel evaluation scheme that ranks detectors in terms of real-world SAR localization requirements. Finally, we propose a novel postprocessing method for robust, approximate object localization: the merging of overlapping bounding boxes (MOB) algorithm. This final processing stage used in the AIR detector significantly improves its performance and usability in the face of real-world aerial SAR missions.

Motivation & Objective

• To automate the slow, error-prone manual inspection of aerial drone footage in land search and rescue (SAR) operations.
• To address the challenge of detecting small, low-contrast persons in high-resolution aerial images with dense object groups.
• To redefine evaluation metrics for aerial person detection to better reflect real-world SAR localization needs, prioritizing detection over precise localization.
• To develop a robust postprocessing method that improves detection reliability in dense person group scenarios common in SAR imagery.
• To achieve state-of-the-art performance in precision and speed while remaining practical for real-time deployment in SAR missions.

Proposed method

• The AIR detector is based on RetinaNet, a one-stage object detector optimized for handling foreground-background class imbalance in aerial imagery.
• The model uses a ResNeXt-101 backbone with feature pyramid network (FPN) for multi-scale feature extraction, enhancing small object detection.
• Online data augmentation, transfer learning, and confidence score threshold calibration are applied to improve generalization and test performance.
• The Merging of Overlapping Bounding boxes (MOB) algorithm postprocesses detection outputs by clustering and merging overlapping boxes, reducing false negatives in dense groups.
• A novel SAR-APD evaluation scheme is introduced, prioritizing detection over localization accuracy, with relaxed IoU thresholds to reflect real-world SAR requirements.
• The method is validated on the HERIDAL benchmark using both VOC2012 and SAR-APD evaluation protocols to compare performance under different criteria.

Experimental results

Research questions

• RQ1 How can deep learning be leveraged to automate aerial person detection in drone-based SAR missions, reducing reliance on manual image inspection?
• RQ2 What model architecture and training strategies yield the best performance for detecting small, low-contrast persons in high-resolution aerial images?
• RQ3 How does the proposed MOB postprocessing method improve detection robustness in dense person group scenarios compared to standard NMS?
• RQ4 To what extent does the new SAR-APD evaluation scheme better reflect real-world operational needs than traditional metrics like VOC2012?
• RQ5 Can the AIR detector outperform human inspectors and existing state-of-the-art models in precision, recall, and inference speed on the HERIDAL benchmark?

Key findings

• The AIR detector achieves 94.9% precision and 92.9% recall on the HERIDAL test set under the proposed SAR-APD evaluation scheme, representing a 21-percentage-point increase in precision compared to prior methods.
• The MOB postprocessing method significantly improves performance by reducing the need for aggressive confidence thresholding, enabling higher recall without sacrificing precision.
• Under the SAR-APD evaluation scheme, AIR achieves 91.7% average precision (AP), outperforming the strongest baseline (Multimodel CNN) in accuracy despite being 10x faster and end-to-end trainable.
• The AIR detector surpasses human performance, which averages 59% precision and 68% recall, with a 33-second average inspection time per image, while AIR processes images in under 1 second.
• The combination of MOB and the SAR-APD evaluation scheme enables a more practical and operationally relevant performance profile, prioritizing detection reliability over exact localization.
• The model demonstrates competitive inference speed (1 second per image) and is suitable for real-time deployment in field SAR operations.