[Paper Review] RoboBEV: Towards Robust Bird's Eye View Perception under Corruptions

Shaoyuan Xie, Lingdong Kong+5 more|arXiv (Cornell University)|Apr 13, 2023

Video Surveillance and Tracking Methods4 citations

TL;DR

This paper introduces RoboBEV, a comprehensive benchmark for evaluating the robustness of camera-based bird's eye view (BEV) perception models under 8 natural corruptions—ranging from environmental (fog, snow) and sensor-induced (motion blur, color quantization) to temporal disruptions (camera crash, frame loss). The study reveals that pre-training and depth-free BEV transformation enhance out-of-distribution robustness, while long-range temporal modeling significantly improves performance under corruption.

ABSTRACT

The recent advances in camera-based bird's eye view (BEV) representation exhibit great potential for in-vehicle 3D perception. Despite the substantial progress achieved on standard benchmarks, the robustness of BEV algorithms has not been thoroughly examined, which is critical for safe operations. To bridge this gap, we introduce RoboBEV, a comprehensive benchmark suite that encompasses eight distinct corruptions, including Bright, Dark, Fog, Snow, Motion Blur, Color Quant, Camera Crash, and Frame Lost. Based on it, we undertake extensive evaluations across a wide range of BEV-based models to understand their resilience and reliability. Our findings indicate a strong correlation between absolute performance on in-distribution and out-of-distribution datasets. Nonetheless, there are considerable variations in relative performance across different approaches. Our experiments further demonstrate that pre-training and depth-free BEV transformation has the potential to enhance out-of-distribution robustness. Additionally, utilizing long and rich temporal information largely helps with robustness. Our findings provide valuable insights for designing future BEV models that can achieve both accuracy and robustness in real-world deployments.

Motivation & Objective

• To address the critical gap in understanding the robustness of camera-based BEV perception models under real-world corruptions.
• To evaluate how existing BEV models perform under diverse natural corruptions that are common in autonomous driving scenarios.
• To identify architectural and training strategies that enhance robustness without relying on in-distribution performance alone.
• To provide actionable insights for designing future BEV models that are both accurate and reliable in safety-critical deployments.

Proposed method

• Proposes RoboBEV, a novel benchmark suite with 8 distinct corruption types: Bright, Dark, Fog, Snow, Motion Blur, Color Quant, Camera Crash, and Frame Lost.
• Applies corruptions across three severity levels to standard BEV detection benchmarks, including nuScenes, to simulate real-world distribution shifts.
• Evaluates 26 state-of-the-art camera-based BEV perception models under both clean and corrupted settings to assess robustness.
• Employs standardized metrics including NuScenes Detection Score (NDS), mAP, and localization errors (mATE, mASE, etc.) for comprehensive evaluation.
• Analyzes the impact of pre-training, depth-free BEV transformation, and temporal modeling by comparing model variants across corruption types.
• Uses radar charts and statistical analysis to visualize performance degradation and identify robustness patterns across models.

Figure 1: The radar charts of existing BEV detectors’ nuScenes Detection Score (NDS) [ 3 ] under eight corruption types. We observe diverse behaviors of different models even with competitive “clean” performance. The NDS is normalized across all the benchmarking BEV models to lie between 0.1 and 1.

Experimental results

Research questions

• RQ1 How do existing BEV models perform under various natural corruptions, and is there a strong correlation between clean-set performance and robustness under corruption?
• RQ2 Which architectural or training components—such as pre-training or depth-free BEV transformation—most significantly improve out-of-distribution robustness?
• RQ3 To what extent does leveraging long and rich temporal information enhance model resilience to temporal corruptions like frame loss or camera crash?
• RQ4 Are there specific corruption types that disproportionately degrade model performance, and do these vary across model architectures?

Key findings

• A strong correlation exists between in-distribution performance and robustness under corruption, but relative robustness does not always follow absolute performance rankings.
• Models using pre-training and depth-free BEV transformation show significantly improved robustness across all corruption types, indicating a key design advantage.
• Longer and richer temporal modeling substantially enhances robustness, especially under temporal corruptions like Frame Lost and Camera Crash.
• Temporal corruptions such as Frame Lost and Camera Crash cause the most severe performance drops, with some models seeing NDS reductions of over 50%.
• Environmental corruptions like Snow and Fog lead to substantial performance degradation, with NDS dropping to as low as 0.11 in extreme cases (e.g., SOLOFusion on Snow).
• Color Quantization and Motion Blur also cause notable performance drops, particularly in mAP and localization error metrics, highlighting sensitivity to visual fidelity.

Figure 2: Corruption examples from the RoboBEV benchmark. Left: Corruption taxonomy. Right: Temporal corruptions. Camera Crash drop fixed set of images along timestamps; Frame Lost randomly drop frames along timestamps. More examples are in Appendix 9.3 .