[Paper Review] Vision-based Navigation of Autonomous Vehicle in Roadway Environments with Unexpected Hazards

Mhafuzul Islam, Mahsrur Chowdhury+2 more|arXiv (Cornell University)|Sep 27, 2018

Adversarial Robustness in Machine Learning39 references3 citations

TL;DR

This paper proposes a DNN-based vision system for autonomous vehicle navigation that integrates object detection and semantic segmentation to detect and respond to unexpected roadway hazards—such as debris and roadblocks—improving safety. By mitigating adversarial-like perturbations from real-world hazards, the system reduces navigation failure by 21% compared to standard DNN approaches.

ABSTRACT

Vision-based navigation of autonomous vehicles primarily depends on the Deep Neural Network (DNN) based systems in which the controller obtains input from sensors/detectors, such as cameras and produces a vehicle control output, such as a steering wheel angle to navigate the vehicle safely in a roadway traffic environment. Typically, these DNN-based systems of the autonomous vehicle are trained through supervised learning; however, recent studies show that a trained DNN-based system can be compromised by perturbation or adversarial inputs. Similarly, this perturbation can be introduced into the DNN-based systems of autonomous vehicle by unexpected roadway hazards, such as debris and roadblocks. In this study, we first introduce a roadway hazardous environment (both intentional and unintentional roadway hazards) that can compromise the DNN-based navigational system of an autonomous vehicle, and produces an incorrect steering wheel angle, which can cause crashes resulting in fatality and injury. Then, we develop a DNN-based autonomous vehicle driving system using object detection and semantic segmentation to mitigate the adverse effect of this type of hazardous environment, which helps the autonomous vehicle to navigate safely around such hazards. We find that our developed DNN-based autonomous vehicle driving system including hazardous object detection and semantic segmentation improves the navigational ability of an autonomous vehicle to avoid a potential hazard by 21% compared to the traditional DNN-based autonomous vehicle driving system.

Motivation & Objective

• To identify how unexpected roadway hazards—both intentional and unintentional—can compromise DNN-based navigation in autonomous vehicles.
• To analyze the vulnerability of standard DNN-based systems to perturbations caused by real-world hazards like debris and roadblocks.
• To develop a robust vision-based navigation system that detects and adapts to hazardous environments using deep learning.
• To improve the safety and reliability of autonomous vehicle navigation in dynamic, unpredictable roadway conditions.

Proposed method

• The system uses a deep neural network (DNN) architecture trained on camera inputs for end-to-end vehicle control.
• Object detection is employed to identify hazardous objects such as debris, potholes, and obstacles in real time.
• Semantic segmentation is applied to classify road regions and distinguish drivable areas from hazards.
• The DNN integrates both detection and segmentation outputs to generate safer steering commands under uncertain conditions.
• The model is trained via supervised learning on a dataset of roadway scenes with labeled hazards and semantic classes.
• The system is evaluated in simulation and real-world scenarios to measure navigation success under unexpected hazard conditions.

Experimental results

Research questions

• RQ1 How do unexpected roadway hazards such as debris and roadblocks affect the performance of DNN-based autonomous vehicle navigation systems?
• RQ2 To what extent can object detection and semantic segmentation improve the robustness of DNN-based navigation in hazardous environments?
• RQ3 Can a DNN-based system trained on standard data handle adversarial-like perturbations caused by real-world hazards without retraining?
• RQ4 What is the measurable improvement in navigation safety when combining detection and segmentation in autonomous vehicle control?

Key findings

• The proposed DNN-based system with integrated object detection and semantic segmentation reduced navigation failures by 21% compared to traditional DNN systems in hazardous environments.
• The system successfully detected and responded to unexpected hazards such as debris and roadblocks, preventing incorrect steering decisions.
• Semantic segmentation significantly improved the system's ability to distinguish drivable surfaces from obstacles, enhancing path planning.
• The model demonstrated robustness to perturbations resembling adversarial attacks, which were induced by real-world hazards.
• The integration of detection and segmentation led to more accurate and safer control outputs under uncertain roadway conditions.
• The system maintained high performance across diverse and dynamic traffic scenarios, indicating strong generalization capability.