[Paper Review] Safe Deep Q-Network for Autonomous Vehicles at Unsignalized Intersection

Kasra Mokhtari, Alan R. Wagner|arXiv (Cornell University)|Jun 8, 2021

Autonomous Vehicle Technology and Safety22 references4 citations

TL;DR

This paper proposes a safe deep reinforcement learning (DRL) framework for autonomous vehicles to navigate unsignalized intersections with pedestrians using two long-short term memory (LSTM) networks to estimate pedestrian trajectories and perceived environment states under noisy observations. A future collision prediction module masks unsafe actions, achieving 0% collisions in both training and unseen intersection topologies while maintaining reasonable speed and efficient navigation in CARLA simulations.

ABSTRACT

We propose a safe DRL approach for autonomous vehicle (AV) navigation through crowds of pedestrians while making a left turn at an unsignalized intersection. Our method uses two long-short term memory (LSTM) models that are trained to generate the perceived state of the environment and the future trajectories of pedestrians given noisy observations of their movement. A future collision prediction algorithm based on the future trajectories of the ego vehicle and pedestrians is used to mask unsafe actions if the system predicts a collision. The performance of our approach is evaluated in two experiments using the high-fidelity CARLA simulation environment. The first experiment tests the performance of our method at intersections that are similar to the training intersection and the second experiment tests our method at intersections with a different topology. For both experiments, our methods do not result in a collision with a pedestrian while still navigating the intersection at a reasonable speed.

Motivation & Objective

• To develop a safe reinforcement learning method for autonomous vehicles navigating crowded, unsignalized intersections where pedestrian behavior is unpredictable.
• To address the challenge of noisy, imperfect sensor observations in real-world autonomous driving scenarios.
• To ensure collision-free navigation while maintaining reasonable speed and task completion rates.
• To evaluate generalization across different intersection topologies after training on a single configuration.
• To integrate safety constraints into deep reinforcement learning without compromising performance or real-time feasibility.

Proposed method

• Two long-short term memory (LSTM) networks are trained to estimate the perceived state of the environment and predict future pedestrian trajectories from noisy observations.
• A future collision prediction algorithm evaluates potential collisions between the ego vehicle and pedestrians using predicted trajectories.
• Unsafe actions are masked in the DRL policy if the system predicts a future collision, enforcing safety during exploration and deployment.
• The DRL agent uses a double deep Q-network (DDQN) with prioritized experience replay (PER) to improve training stability and sample efficiency.
• A conditional reward function is designed to encourage safe, timely navigation while penalizing collisions and excessive delays.
• The framework is evaluated in the high-fidelity CARLA simulation environment under realistic sensor noise and diverse intersection geometries.

Experimental results

Research questions

• RQ1 Can a DRL-based control policy achieve zero collisions during autonomous vehicle navigation at unsignalized intersections with dense pedestrian traffic?
• RQ2 How does the integration of LSTM-based perception and future collision detection improve safety under noisy observations compared to standard DRL?
• RQ3 To what extent can a policy trained on one intersection topology generalize to a different, unseen intersection layout?
• RQ4 How does the proposed safe DRL method compare to rule-based baselines in terms of collision rate, speed, and navigation efficiency?
• RQ5 What role does the belief update LSTM play in improving robustness to sensor noise in real-time decision-making?

Key findings

• The proposed safe DRL method achieved 0% collision episodes in both experiments, outperforming the rule-based baseline and standard DRL agents.
• The SRL agent completed navigation in 28.12s (Experiment 1) and 28.61s (Experiment 2), with an average speed of 3.25m/s and 3.27m/s, respectively.
• The SRL agent maintained an average distance of 7.19m and 7.21m from the closest pedestrian, indicating safer interaction behavior.
• The future collision detector and SRL agents achieved zero collisions, while the standard RL agent had a 6% collision rate due to noisy observations.
• The belief update LSTM significantly improved robustness to perception noise, as the pure RL agent without this module had a 6% collision rate compared to 0% in the SRL version.
• The method generalized well across different intersection topologies, maintaining consistent performance in both training and unseen environments.