[Paper Review] Comparison of Object Detection Algorithms for Street-level Objects

Martinus Grady Naftali, Jason Sebastian Sulistyawan+1 more|arXiv (Cornell University)|Aug 24, 2022

Advanced Neural Network Applications4 citations

TL;DR

This study evaluates five one-stage object detection algorithms—SSD MobileNetv2 FPN-lite, YOLOv3, YOLOv4, YOLOv5l, and YOLOv5s—on a modified Udacity self-driving car dataset for street-level object detection. YOLOv5l achieved the highest accuracy (mAP@.5: 0.593), while SSD MobileNetv2 FPN-lite was fastest (3.20ms inference time), and YOLOv5s offered the best balance of speed and accuracy for real-time applications like autonomous vehicles.

ABSTRACT

Object detection for street-level objects can be applied to various use cases, from car and traffic detection to the self-driving car system. Therefore, finding the best object detection algorithm is essential to apply it effectively. Many object detection algorithms have been released, and many have compared object detection algorithms, but few have compared the latest algorithms, such as YOLOv5, primarily which focus on street-level objects. This paper compares various one-stage detector algorithms; SSD MobileNetv2 FPN-lite 320x320, YOLOv3, YOLOv4, YOLOv5l, and YOLOv5s for street-level object detection within real-time images. The experiment utilizes a modified Udacity Self Driving Car Dataset with 3,169 images. Dataset is split into train, validation, and test; Then, it is preprocessed and augmented using rescaling, hue shifting, and noise. Each algorithm is then trained and evaluated. Based on the experiments, the algorithms have produced decent results according to the inference time and the values of their precision, recall, F1-Score, and Mean Average Precision (mAP). The results also shows that YOLOv5l outperforms the other algorithms in terms of accuracy with a mAP@.5 of 0.593, MobileNetv2 FPN-lite has the fastest inference time among the others with only 3.20ms inference time. It is also found that YOLOv5s is the most efficient, with it having a YOLOv5l accuracy and a speed almost as quick as the MobileNetv2 FPN-lite. This shows that various algorithm are suitable for street-level object detection and viable enough to be used in self-driving car.

Motivation & Objective

• To evaluate the performance of modern one-stage object detection algorithms on street-level object detection tasks.
• To compare inference speed, precision, recall, F1-score, and mAP across YOLOv3, YOLOv4, YOLOv5l, YOLOv5s, and SSD MobileNetv2 FPN-lite.
• To identify the most suitable algorithm for real-time applications such as self-driving cars.
• To assess the impact of data augmentation (rescaling, hue shifting, noise) on model generalization and robustness.
• To provide empirical benchmarks for the latest YOLOv5 variants in comparison to prior models on a real-world driving dataset.

Proposed method

• Trained and evaluated five one-stage detectors: SSD MobileNetv2 FPN-lite, YOLOv3, YOLOv4, YOLOv5l, and YOLOv5s on a modified Udacity Self-Driving Car dataset with 3,169 images.
• Split the dataset into training, validation, and test sets using standard 70/15/15 split.
• Applied data augmentation techniques: image rescaling, hue shifting, and noise injection to improve model generalization.
• Measured performance using precision, recall, F1-score, mAP@.5, mAP@.5:95, and inference time on a system with an 8th Gen Intel i5-8600K CPU, GTX 1050 Ti GPU, and 16GB RAM.
• Used Intersection over Union (IoU) thresholds of 0.5 and 0.5–0.95 to compute mAP, with mAP@.5:95 representing average precision across multiple IoU thresholds.
• Evaluated models using standard object detection metrics: Precision = TP / (TP + FP), Recall = TP / (TP + FN), F1-Score = 2 × (Precision × Recall) / (Precision + Recall).

Experimental results

Research questions

• RQ1 Which one-stage object detection algorithm achieves the highest mAP for street-level object detection on the modified Udacity dataset?
• RQ2 Which algorithm offers the fastest inference time suitable for real-time deployment in autonomous vehicles?
• RQ3 How do the newer YOLOv5 models (YOLOv5l and YOLOv5s) compare in accuracy and speed to YOLOv3 and YOLOv4 on the same benchmark?
• RQ4 To what extent does data augmentation improve detection performance across different models?
• RQ5 Can YOLOv5s achieve a favorable trade-off between accuracy and speed, making it ideal for real-time self-driving applications?

Key findings

• YOLOv5l achieved the highest mAP@.5 score of 0.593, outperforming all other models in accuracy.
• SSD MobileNetv2 FPN-lite recorded the fastest inference time at 3.20ms, making it the most efficient for real-time inference.
• YOLOv5s achieved a mAP@.5 of 0.530, which is only 11% lower than YOLOv5l’s 0.593, while maintaining a fast inference time of 8.50ms.
• YOLOv4 performed worse than YOLOv5l and YOLOv3 in terms of F1-score, precision, and mAP, with the slowest inference time at 27.90ms.
• YOLOv5s demonstrated a strong balance between accuracy and speed, making it the most suitable model for real-time self-driving car applications.
• SSD MobileNetv2 FPN-lite had the lowest mAP@.5 (0.315), indicating poor detection accuracy despite its speed advantage.