[Paper Review] On the Robustness of Object Detection Models on Aerial Images

Haodong He, Jian Ding+2 more|arXiv (Cornell University)|Aug 29, 2023

Advanced Neural Network ApplicationsComputer Science3 citations

TL;DR

This paper introduces two novel benchmarks for evaluating the robustness of object detection models in aerial images: one with 19 common corruptions and another specifically targeting cloud corruption, which is prevalent in aerial but rare in natural images. The study reveals that larger models, improved backbone architectures (e.g., ConvNeXt-T, Swin-L), rotation-invariant modules (RiRoI Align), and strategic data augmentation significantly enhance robustness, especially under cloud-induced corruptions and out-of-distribution conditions.

ABSTRACT

The robustness of object detection models is a major concern when applied to real-world scenarios. The performance of most models tends to degrade when confronted with images affected by corruptions, since they are usually trained and evaluated on clean datasets. While numerous studies have explored the robustness of object detection models on natural images, there is a paucity of research focused on models applied to aerial images, which feature complex backgrounds, substantial variations in scales, and orientations of objects. This paper addresses the challenge of assessing the robustness of object detection models on aerial images, with a specific emphasis on scenarios where images are affected by clouds. In this study, we introduce two novel benchmarks based on DOTA-v1.0. The first benchmark encompasses 19 prevalent corruptions, while the second focuses on the cloud-corrupted condition-a phenomenon uncommon in natural images yet frequent in aerial photography. We systematically evaluate the robustness of mainstream object detection models and perform necessary ablation experiments. Through our investigations, we find that rotation-invariant modeling and enhanced backbone architectures can improve the robustness of models. Furthermore, increasing the capacity of Transformer-based backbones can strengthen their robustness. The benchmarks we propose and our comprehensive experimental analyses can facilitate research on robust object detection on aerial images. The codes and datasets are available at: https://github.com/hehaodong530/DOTA-C.

Motivation & Objective

• To address the lack of standardized benchmarks for evaluating object detection model robustness in aerial images, particularly under real-world corruptions.
• To investigate how common image corruptions and cloud-specific distortions affect the performance of state-of-the-art object detection models on aerial datasets.
• To identify architectural and training strategy factors that enhance robustness in aerial object detection under distribution shift.
• To provide a comprehensive evaluation of mainstream models on out-of-distribution data, including unseen corruption types.
• To offer actionable insights for designing more robust aerial object detectors through ablation studies on backbone choice, model capacity, and data augmentation.

Proposed method

• Constructed a corruption benchmark based on DOTA-v1.0 by applying 19 standard corruptions from ImageNet-C across five severity levels.
• Developed a novel cloud corruption benchmark by transferring real atmospheric clouds from authentic satellite images to clean aerial images for realistic simulation.
• Trained and evaluated multiple state-of-the-art object detection models (e.g., RoI Transformer, ReDet) on both clean and corrupted test sets without exposure to corruptions during training.
• Conducted ablation studies on backbone architecture (ResNet, ConvNeXt, Swin), model capacity (parameter count), and modules (RiRoI Align), as well as data augmentation strategies (RandomRotate, Mosaic).
• Used metrics including mAP@50, rPC (relative performance drop), and $ PC_{\text{clouds}}$ to quantify robustness across corruption types and severity levels.
• Performed comparative analysis across corruption categories (noise, blur, weather, digital) to assess model resilience under different degradation types.

Experimental results

Research questions

• RQ1 How do mainstream object detection models perform on aerial images corrupted by common image distortions, such as brightness, snow, and blur?
• RQ2 To what extent does cloud corruption—common in aerial imagery but rare in natural images—affect the detection performance of state-of-the-art models?
• RQ3 Which architectural components (e.g., backbone, attention modules) and training strategies (e.g., data augmentation) most effectively improve robustness under out-of-distribution corruptions?
• RQ4 How does increasing model capacity (e.g., from Swin-T to Swin-L) influence robustness across different corruption types?
• RQ5 Are data augmentation techniques like RandomRotate and Mosaic consistently beneficial for improving robustness, or do their effects vary by corruption type?

Key findings

• The performance of all evaluated models, including RoI Transformer and ReDet, degrades significantly on corrupted aerial images, with mAP@50 dropping by up to 40% on average under severe corruptions.
• Replacing the backbone with ConvNeXt-T or Swin-L led to the most substantial improvements in robustness, with rPC values increasing by over 15 percentage points on average across corruption types.
• The RiRoI Align module significantly enhanced robustness on blur corruptions, improving rPC by over 6% compared to standard RoI Align.
• Larger models (e.g., Swin-L) consistently outperformed smaller variants (e.g., Swin-T) across all corruption types, with rPC increasing by up to 17% on weather corruptions.
• Data augmentation strategies like RandomRotate and Mosaic showed inconsistent benefits—Mosaic even reduced performance on digital corruptions, indicating limited generalization for robustness.
• The model with ConvNeXt-T backbone achieved the highest rPC on noise (47.17%) and weather (74.82%) corruptions, while ReDet achieved the best rPC on blur corruptions (56.77%), highlighting the importance of architecture-specific design.