[Paper Review] Lane Change Decision-Making through Deep Reinforcement Learning

Mukesh Ghimire, Malobika Roy Choudhury+1 more|arXiv (Cornell University)|Dec 24, 2021

Autonomous Vehicle Technology and Safety4 citations

TL;DR

This paper proposes a rule-based Deep Q-Network (DQN) for autonomous lane change decision-making in dynamic traffic, combining deep reinforcement learning with safety constraints. The method achieves a 0.8 safety rate and 47 MPH average speed, outperforming standard DQN by reducing unnecessary lane changes and improving collision avoidance.

ABSTRACT

Due to the complexity and volatility of the traffic environment, decision-making in autonomous driving is a significantly hard problem. In this project, we use a Deep Q-Network, along with rule-based constraints to make lane-changing decision. A safe and efficient lane change behavior may be obtained by combining high-level lateral decision-making with low-level rule-based trajectory monitoring. The agent is anticipated to perform appropriate lane-change maneuvers in a real-world-like udacity simulator after training it for a total of 100 episodes. The results shows that the rule-based DQN performs better than the DQN method. The rule-based DQN achieves a safety rate of 0.8 and average speed of 47 MPH

Motivation & Objective

• To develop a robust lane change decision-making system for autonomous vehicles under complex, dynamic traffic conditions.
• To improve safety and efficiency in lane changes by integrating rule-based constraints with deep Q-learning.
• To replicate and validate the results of prior work on DQN-based lane changing in a simulated environment.
• To evaluate the impact of rule-based constraints on DQN performance in terms of safety, speed, and lane change frequency.

Proposed method

• The approach uses a Deep Q-Network (DQN) trained via reinforcement learning to learn optimal lane change policies.
• Rule-based constraints are integrated to enforce safety, such as maintaining minimum distance to surrounding vehicles and avoiding collisions.
• The state space includes vehicle speed, relative positions of surrounding vehicles, and lane information, represented as a 1D vector.
• The action space consists of discrete choices: change to left lane, change to right lane, or maintain current lane.
• A reward function is designed to encourage safe and efficient driving, penalizing collisions and rewarding proximity to lane center and consistent speed.
• Training is conducted over 100 episodes in the Udacity self-driving car simulator, with evaluation on 10 test episodes.

Experimental results

Research questions

• RQ1 How does integrating rule-based safety constraints improve the performance of a DQN agent in autonomous lane change decision-making?
• RQ2 What is the impact of rule-based constraints on the frequency of lane changes and overall safety rate?
• RQ3 How does the rule-based DQN compare to a standard DQN in terms of average speed, safety rate, and number of lane changes?
• RQ4 Can the agent generalize to real-world-like traffic conditions while maintaining safety and efficiency?

Key findings

• The rule-based DQN achieved a safety rate of 0.8, significantly outperforming the standard DQN, which had a safety rate of 0.2.
• The rule-based DQN reduced the average number of lane changes to 8.80 per episode, compared to 36.20 for the standard DQN, indicating more selective and efficient behavior.
• The rule-based DQN maintained an average speed of 47 MPH, slightly higher than the 46 MPH achieved by the standard DQN.
• During training, lane change frequency decreased from an average of 64.1 per episode to 10.0 in testing, indicating the agent learned to change lanes only when necessary.
• The agent exhibited erratic behavior during early training due to exploration, but stabilized over time, showing effective policy learning.
• The study observed that minor collisions could trigger cascading accidents, highlighting the need for refined reward shaping to penalize such events more strongly.