[Paper Review] Road Damage Detection Using Deep Neural Networks with Images Captured Through a Smartphone

Hiroya Maeda, Yoshihide Sekimoto+3 more|arXiv (Cornell University)|Jan 29, 2018

Infrastructure Maintenance and MonitoringEngineering26 references248 citations

TL;DR

The authors create a large-scale road damage dataset captured with a smartphone, train SSD-based detectors to classify eight damage types, and show real-time smartphone inference with public datasets and code.

ABSTRACT

Research on damage detection of road surfaces using image processing techniques has been actively conducted, achieving considerably high detection accuracies. Many studies only focus on the detection of the presence or absence of damage. However, in a real-world scenario, when the road managers from a governing body need to repair such damage, they need to clearly understand the type of damage in order to take effective action. In addition, in many of these previous studies, the researchers acquire their own data using different methods. Hence, there is no uniform road damage dataset available openly, leading to the absence of a benchmark for road damage detection. This study makes three contributions to address these issues. First, to the best of our knowledge, for the first time, a large-scale road damage dataset is prepared. This dataset is composed of 9,053 road damage images captured with a smartphone installed on a car, with 15,435 instances of road surface damage included in these road images. In order to generate this dataset, we cooperated with 7 municipalities in Japan and acquired road images for more than 40 hours. These images were captured in a wide variety of weather and illuminance conditions. In each image, we annotated the bounding box representing the location and type of damage. Next, we used a state-of-the-art object detection method using convolutional neural networks to train the damage detection model with our dataset, and compared the accuracy and runtime speed on both, using a GPU server and a smartphone. Finally, we demonstrate that the type of damage can be classified into eight types with high accuracy by applying the proposed object detection method. The road damage dataset, our experimental results, and the developed smartphone application used in this study are publicly available (https://github.com/sekilab/RoadDamageDetector/).

Motivation & Objective

• Motivate practical road damage assessment by requiring type-specific damage classification for actionable maintenance.
• Create a uniform, publicly available dataset of road damage images captured from a car-mounted smartphone under diverse conditions.
• Evaluate state-of-the-art deep object detectors on road damage detection and classification tasks.
• Demonstrate that damage type can be identified with high accuracy using end-to-end deep learning on a mobile-friendly platform.

Proposed method

• Develop a large-scale dataset of 9,053 annotated road-damage images (15,435 damage instances) captured with a smartphone mounted in a car.
• Train and evaluate SSD-based object detectors (SSD Inception V2 and SSD MobileNet) on the dataset.
• Use 600x600 images, resizing to 300x300 for SSD inputs, with learning-rate schedules as specified (Inception V2: 0.002, decay 0.95 every 10k iterations; MobileNet: 0.003, decay 0.95 every 10k iterations).
• Assess performance with IOU threshold of 0.5 and report class-wise recall, precision, and accuracy.
• Analyze speed: GPU-based inferences vs. smartphone-based inferences (MobileNet faster; smartphone ~1500 ms).
• Provide publicly available trained models, code, and a smartphone app for real-time detection.

Experimental results

Research questions

• RQ1 Can end-to-end deep learning object detectors accurately classify eight types of road damage from smartphone-captured images?
• RQ2 Is a large, publicly available smartphone-based road damage dataset feasible and useful for benchmarking?
• RQ3 What are the detection performance (recall/precision) and inference speeds of SSD-based detectors on this dataset, including on-device smartphone inference?

Key findings

• A dataset of 9,053 annotated road-damage images with 15,435 damage instances was created and released.
• SSD MobileNet achieved better results than SSD Inception V2 across classes, with significant on-device inference capability.
• The best-performing configurations achieved recalls and precisions greater than 75% for some damage types.
• Smartphone-based detection can run in about 1.5 seconds per image, enabling real-time roadside assessment in moving vehicles.
• Inference on a smartphone matches server-based accuracy for detected classes, enabling practical field deployment.