[Paper Review] Challenging Environments for Traffic Sign Detection: Reliability Assessment under Inclement Conditions

Dogancan Temel, Tariq Alshawi+2 more|arXiv (Cornell University)|Feb 19, 2019

Infrastructure Maintenance and Monitoring47 references20 citations

TL;DR

This paper introduces the CURE-TSD dataset, a video-based traffic sign detection benchmark with realistic challenging conditions such as rain, snow, haze, and low light. It evaluates state-of-the-art deep learning models under these conditions, revealing significant performance drops—average precision loss of 0.17 and recall loss of 0.28 under severe conditions—highlighting the urgent need for more robust algorithms in autonomous driving systems.

ABSTRACT

State-of-the-art algorithms successfully localize and recognize traffic signs over existing datasets, which are limited in terms of challenging condition type and severity. Therefore, it is not possible to estimate the performance of traffic sign detection algorithms under overlooked challenging conditions. Another shortcoming of existing datasets is the limited utilization of temporal information and the unavailability of consecutive frames and annotations. To overcome these shortcomings, we generated the CURE-TSD video dataset and hosted the first IEEE Video and Image Processing (VIP) Cup within the IEEE Signal Processing Society. In this paper, we provide a detailed description of the CURE-TSD dataset, analyze the characteristics of the top performing algorithms, and provide a performance benchmark. Moreover, we investigate the robustness of the benchmarked algorithms with respect to sign size, challenge type and severity. Benchmarked algorithms are based on state-of-the-art and custom convolutional neural networks that achieved a precision of 0.55 and a recall of 0.32, F0.5 score of 0.48 and F2 score of 0.35. Experimental results show that benchmarked algorithms are highly sensitive to tested challenging conditions, which result in an average performance drop of 0.17 in terms of precision and a performance drop of 0.28 in recall under severe conditions. The dataset is publicly available at https://github.com/olivesgatech/CURE-TSD.

Motivation & Objective

• To address the lack of comprehensive, real-world challenging conditions in existing traffic sign detection datasets.
• To provide a benchmark for evaluating the robustness of traffic sign detection algorithms under diverse environmental degradations.
• To analyze the impact of challenge type, severity, and sign size on algorithmic performance.
• To investigate the correlation between performance in simulated (synthesized) environments and real-world conditions.
• To promote the development of more robust deep learning models for autonomous vehicle perception under adverse weather and lighting.

Proposed method

• Generated the CURE-TSD video dataset with 100+ sequences capturing traffic signs under real-world challenging conditions including rain, snow, haze, low light, and overexposure.
• Hosted the IEEE Video and Image Processing Cup 2017 to benchmark top-performing detection algorithms on the CURE-TSD dataset.
• Evaluated four top-performing teams' models using standard metrics: precision, recall, F0.5, and F2 scores across varying challenge types and severities.
• Classified challenges by severity using a size-based metric (x-axis), where smaller signs indicate higher challenge level, and analyzed performance trends.
• Split the test set into real-world and synthesized data to compute Spearman correlation between performance in both domains.
• Analyzed algorithmic robustness by comparing performance across challenge types (e.g., haze, rain, codec errors) and sign sizes.

Experimental results

Research questions

• RQ1 How do state-of-the-art traffic sign detection models perform under realistic, severe environmental challenges such as rain, snow, and low light?
• RQ2 What is the relationship between sign size and detection performance under challenging conditions?
• RQ3 How does the performance of detection algorithms vary across different types and levels of environmental degradation?
• RQ4 To what extent can performance in simulated (synthesized) environments predict real-world robustness under similar conditions?
• RQ5 Which algorithmic architectures demonstrate the highest robustness to environmental degradation in traffic sign detection?

Key findings

• The top-performing algorithms achieved a precision of 0.55 and recall of 0.32, with an F0.5 score of 0.48 and F2 score of 0.35 on the CURE-TSD benchmark.
• Under severe challenging conditions, the average precision dropped by 0.17 and recall by 0.28, indicating high sensitivity to environmental degradation.
• Team Neurons achieved the highest performance across all sign size levels, while Team IIP showed the most significant performance degradation, especially due to codec errors and rain.
• Performance variation was most pronounced in Team IIP and Team Markovians, with Team Markovians showing low variation in recall and F2 but poor performance in haze conditions.
• The Spearman correlation between real-world and synthesized environment performance was 0.915 for recall and 0.781 for precision, indicating strong predictive power of simulation-based robustness testing.
• The study demonstrates that current deep learning models are highly vulnerable to environmental challenges, underscoring the need for more robust algorithmic designs in autonomous vehicle perception systems.