[论文解读] Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming

Claudio Michaelis, Benjamin Mitzkus+6 more|arXiv (Cornell University)|Jul 17, 2019

Advanced Neural Network Applications被引用 199

一句话总结

论文引入 Robust Detection Benchmark (PASCAL-C, COCO-C, Cityscapes-C) 以评估目标检测模型在多样图像腐蚀下的表现，并显示通过对训练数据进行风格化可以提升对腐蚀和数据集的鲁棒性。

ABSTRACT

The ability to detect objects regardless of image distortions or weather conditions is crucial for real-world applications of deep learning like autonomous driving. We here provide an easy-to-use benchmark to assess how object detection models perform when image quality degrades. The three resulting benchmark datasets, termed Pascal-C, Coco-C and Cityscapes-C, contain a large variety of image corruptions. We show that a range of standard object detection models suffer a severe performance loss on corrupted images (down to 30--60\\% of the original performance). However, a simple data augmentation trick---stylizing the training images---leads to a substantial increase in robustness across corruption type, severity and dataset. We envision our comprehensive benchmark to track future progress towards building robust object detection models. Benchmark, code and data are publicly available.

研究动机与目标

• Motivate robust object detection for autonomous driving under varying weather and image distortions.
• Provide an easy-to-use, standardized benchmark to quantify robustness of detection models.
• Show how synthetic corruptions relate to real-world distortions like rain, snow, and fog.
• Demonstrate that data augmentation via stylized training can substantially improve robustness.

提出的方法

• Propose Robust Detection Benchmark with three datasets (PASCAL-C, COCO-C, Cityscapes-C) each containing 15 corruptions across five severity levels.
• Adopt corruption types and evaluation metrics inspired by ImageNet-C, using mean performance under corruption (mPC) and relative degradation (rPC).
• Evaluate standard object detection models (e.g., Faster R-CNN, Mask R-CNN, Cascade variants, RetinaNet, Hybrid Task Cascade) with various backbones.
• Introduce style transfer as data augmentation (stylized training data) to reduce texture bias and improve robustness.
• Compare training regimes: standard data, stylized data, and combined datasets to assess robustness gains.
• Provide open-source code and prebuilt tools for corruptions (imagecorruptions), stylization (stylize-datasets), and benchmark orchestration (robust-detection-benchmark).

实验结果

研究问题

• RQ1 How do common object detection models perform on corrupted images across VOC, COCO, and Cityscapes datasets?
• RQ2 Can a simple data augmentation—stylizing training images—improve robustness to a broad set of corruptions without sacrificing clean performance?
• RQ3 Does robustness to synthetic corruptions translate to improved robustness to natural distortions like rain, snow, and fog in driving scenarios?
• RQ4 What is the relationship between model backbone capacity and corruption robustness?
• RQ5 Is stylized pretraining or stylized final training more effective for corruption robustness in object detection?

主要发现

• Many object detection and instance segmentation models suffer substantial performance declines on corrupted images.
• Stronger backbones generally improve robustness to corruptions, indicating robustness scales with image encoding capacity.
• Training on stylized data significantly improves robustness across corruptions, and combining stylized with standard data yields best overall performance on corrupted data with minimal loss on clean data.
• Stylized training also enhances generalization to natural distortions such as rain, snow, and fog, and helps with day/night and foggy conditions in driving datasets.
• The Robust Detection Benchmark (PASCAL-C, COCO-C, Cityscapes-C) provides a standardized way to track progress in robustness and reveals that there is still room for architectural and algorithmic advances beyond stylization.