[Paper Review] Unmanned Aerial Vehicle Visual Detection and Tracking using Deep Neural Networks: A Performance Benchmark

Brian K. S. Isaac-Medina, Matt Poyser+4 more|arXiv (Cornell University)|Mar 25, 2021

Video Surveillance and Tracking Methods51 references95 citations

TL;DR

This paper presents the first comprehensive benchmark for deep learning-based UAV detection and tracking using visible and infrared imagery, evaluating four object detectors and three trackers across three diverse datasets. It achieves state-of-the-art performance with a 98.6% mAP in detection and 98.7% MOTA in tracking, demonstrating cross-modality transferability from infrared to visible spectra with 82.8% mAP.

ABSTRACT

Unmanned Aerial Vehicles (UAV) can pose a major risk for aviation safety, due to both negligent and malicious use. For this reason, the automated detection and tracking of UAV is a fundamental task in aerial security systems. Common technologies for UAV detection include visible-band and thermal infrared imaging, radio frequency and radar. Recent advances in deep neural networks (DNNs) for image-based object detection open the possibility to use visual information for this detection and tracking task. Furthermore, these detection architectures can be implemented as backbones for visual tracking systems, thereby enabling persistent tracking of UAV incursions. To date, no comprehensive performance benchmark exists that applies DNNs to visible-band imagery for UAV detection and tracking. To this end, three datasets with varied environmental conditions for UAV detection and tracking, comprising a total of 241 videos (331,486 images), are assessed using four detection architectures and three tracking frameworks. The best performing detector architecture obtains an mAP of 98.6% and the best performing tracking framework obtains a MOTA of 96.3%. Cross-modality evaluation is carried out between visible and infrared spectrums, achieving a maximal 82.8% mAP on visible images when training in the infrared modality. These results provide the first public multi-approach benchmark for state-of-the-art deep learning-based methods and give insight into which detection and tracking architectures are effective in the UAV domain.

Motivation & Objective

• To establish a standardized, multi-dataset benchmark for UAV detection and tracking using deep neural networks.
• To evaluate the performance of leading object detection and tracking architectures under diverse environmental and imaging conditions.
• To investigate cross-modality transfer learning between infrared and visible-band UAV imagery.
• To identify the most robust detection and tracking frameworks for counter-UAV applications in real-world scenarios.
• To provide a public benchmark toolkit to accelerate research in automated UAV detection and tracking.

Proposed method

• The study evaluates four object detection architectures—Faster R-CNN, YOLOv3, SSD512, and DETR—on three UAV datasets: MAV-VID, Drone-vs-Bird, and Anti-UAV.
• Three tracking frameworks—SORT, DeepSORT, and Tracktor—are evaluated using the detected bounding boxes as input for temporal association.
• Cross-modality evaluation is performed by training detectors on infrared data and testing on visible-band images, and vice versa.
• The benchmark uses standard metrics: mean Average Precision (mAP) for detection and Multiple Object Tracking Accuracy (MOTA) for tracking.
• Datasets include both ground-based and UAV-mounted cameras, capturing UAVs at varying distances, in dynamic scenes, and under optical and thermal infrared conditions.
• The evaluation is conducted across multiple environmental conditions, including complex backgrounds, high-speed camera motion, and occlusions.

Experimental results

Research questions

• RQ1 Which deep learning object detection architecture achieves the highest mAP for UAVs in visible-band imagery across diverse environmental conditions?
• RQ2 How does cross-modality training (e.g., infrared to visible) affect UAV detection performance?
• RQ3 Which tracking framework achieves the highest MOTA when tracking small, fast-moving UAVs under challenging visual conditions?
• RQ4 How do camera motion and background complexity impact the performance of tracking systems?
• RQ5 To what extent can generic object detectors be effectively adapted for UAV-specific detection and tracking tasks?

Key findings

• The best-performing detector, YOLOv3, achieves a mean Average Precision (mAP) of 98.6% on visible-band UAV imagery.
• Faster R-CNN achieves the highest mAP (up to 0.770) for small UAVs, indicating superior performance for early detection.
• The Tracktor tracking framework achieves the highest MOTA of 98.7%, demonstrating its effectiveness in handling high-speed camera motion and long-term tracking.
• Cross-modality detection via training on infrared data and testing on visible images achieves 82.8% mAP, proving effective transferability between modalities.
• DETR-based detection backbones yield strong performance (mAP > 0.94) and are well-suited for small-object tracking in counter-UAV systems.
• The re-identification network in DeepSORT and Tracktor does not consistently improve tracking performance and can even degrade it, indicating a need for UAV-specific re-identification models.