[Paper Review] Leveraging Synthetic Data in Object Detection on Unmanned Aerial Vehicles

Benjamin Kiefer, David J. Ott+1 more|arXiv (Cornell University)|Dec 22, 2021

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes leveraging high-resolution synthetic data from Grand Theft Auto V (GTAV) via an extended DeepGTAV framework to improve object detection on unmanned aerial vehicles (UAVs). By generating large-scale, metadata-aligned synthetic datasets across diverse UAV scenarios and applying synthetic pre-training, the approach achieves performance on par with real-data-only training, with key gains from graphics quality and metadata alignment—especially when targeting specific camera angles or time-of-day distributions.

ABSTRACT

Acquiring data to train deep learning-based object detectors on Unmanned Aerial Vehicles (UAVs) is expensive, time-consuming and may even be prohibited by law in specific environments. On the other hand, synthetic data is fast and cheap to access. In this work, we explore the potential use of synthetic data in object detection from UAVs across various application environments. For that, we extend the open-source framework DeepGTAV to work for UAV scenarios. We capture various large-scale high-resolution synthetic data sets in several domains to demonstrate their use in real-world object detection from UAVs by analyzing multiple training strategies across several models. Furthermore, we analyze several different data generation and sampling parameters to provide actionable engineering advice for further scientific research. The DeepGTAV framework is available at https://git.io/Jyf5j.

Motivation & Objective

• Address the scarcity and high cost of real-world UAV object detection datasets, which are hindered by legal restrictions, privacy concerns, and complex data collection.
• Overcome domain shift and performance gaps in UAV object detection by leveraging synthetic data generation as a scalable alternative.
• Improve transferability of synthetic data to real-world UAV applications through targeted data generation and metadata alignment.
• Provide actionable engineering insights on synthetic data generation parameters—such as graphics quality, metadata alignment, and data sampling strategies—for optimal model performance.
• Demonstrate that synthetic pre-training can match or exceed real-data-only training performance when combined with domain-adaptive data curation.

Proposed method

• Extended the DeepGTAV framework to support airborne camera configurations, including dynamic agent and camera positioning, rotation, and environment manipulation in GTAV.
• Generated three large-scale, high-resolution (4K) synthetic object detection datasets across distinct UAV application domains: traffic surveillance (VisDrone), maritime search (SeaDronesSee), and livestock monitoring (Cattle).
• Automated metadata extraction and alignment by sampling synthetic data distributions (e.g., time of day, camera angle) to match real-world data distributions.
• Trained YOLOv5 and EfficientDet-D0 models using both synthetic-only and synthetic-to-real transfer learning strategies on the generated datasets.
• Conducted ablation studies on graphics quality (high vs. low), data volume, pre-training on ImageNet, and metadata alignment to evaluate their impact on mAP@50.
• Used bootstrap sampling and distribution matching to create synthetic subsets that mirror real-world data characteristics, improving data efficiency and model generalization.

Figure 1: Real (left) and synthetic (right) image samples in different applications scenarios with ground truth annotations. Representative objects are magnified.

Experimental results

Research questions

• RQ1 Can synthetic data from a video game engine like GTAV effectively improve object detection performance on real UAV imagery?
• RQ2 How does the graphics quality of the simulation engine affect the transferability of synthetic data to real-world UAV detection tasks?
• RQ3 To what extent does aligning synthetic data distributions (e.g., time of day, camera angle) with real-world data improve model performance?
• RQ4 Does synthetic pre-training outperform or match real-data-only training in UAV object detection, and under what conditions?
• RQ5 What role does pre-training on large-scale natural image datasets play when combined with synthetic data for UAV detection?

Key findings

• Synthetic pre-training with high-quality GTAV data improved mAP@50 from 43.9 (real-only baseline) to 45.1 when fine-tuned on VisDrone, demonstrating effective domain transfer.
• High-quality graphics in synthetic data led to a 1.8% mAP@50 improvement over low-quality settings in purely synthetic training (10.2 vs. 8.4), though the effect diminished in transfer learning.
• Aligning synthetic data to real-world time-of-day distributions increased synthetic-only mAP@50 by 4.1 points (from 10.5 to 14.6) on SeaDronesSee, with minimal gain in transfer learning.
• Restricting synthetic data to only downward-facing camera angles (within 20° of real Cattle dataset) improved synthetic-only mAP@50 from 29.2 to 36.6, showing that targeted data curation enhances efficiency.
• Synthetic-to-real transfer learning achieved 85.8 mAP@50 on Cattle and 60.5 mAP@50 on SeaDronesSee, matching or exceeding real-only baselines in some cases.
• Pre-training on ImageNet consistently improved performance across all settings, with no observed interactive effect with synthetic data, indicating complementary benefits.

Figure 2: The effect of the real and synthetic data set size on the model performance for VisDrone and SeaDronesSee. The color shading specifies the size of the synthetic data set that was used for pre-training.