[Paper Review] Research on Autonomous Driving Decision-making Strategies based Deep Reinforcement Learning

Zixiang Wang, Hao Yan+3 more|arXiv (Cornell University)|Aug 6, 2024

Traffic Prediction and Management Techniques4 citations

TL;DR

This paper proposes a deep reinforcement learning (DRL)-based decision-making framework for autonomous driving, comparing Deep Q-Network (DQN) and Proximal Policy Optimization (PPO) to learn adaptive driving strategies in complex traffic environments. The improved reward function enhances robustness, and experiments show DRL methods outperform rule-based systems across multiple driving tasks.

ABSTRACT

The behavior decision-making subsystem is a key component of the autonomous driving system, which reflects the decision-making ability of the vehicle and the driver, and is an important symbol of the high-level intelligence of the vehicle. However, the existing rule-based decision-making schemes are limited by the prior knowledge of designers, and it is difficult to cope with complex and changeable traffic scenarios. In this work, an advanced deep reinforcement learning model is adopted, which can autonomously learn and optimize driving strategies in a complex and changeable traffic environment by modeling the driving decision-making process as a reinforcement learning problem. Specifically, we used Deep Q-Network (DQN) and Proximal Policy Optimization (PPO) for comparative experiments. DQN guides the agent to choose the best action by approximating the state-action value function, while PPO improves the decision-making quality by optimizing the policy function. We also introduce improvements in the design of the reward function to promote the robustness and adaptability of the model in real-world driving situations. Experimental results show that the decision-making strategy based on deep reinforcement learning has better performance than the traditional rule-based method in a variety of driving tasks.

Motivation & Objective

• To address the limitations of rule-based decision-making in handling complex, dynamic traffic scenarios.
• To develop a learning-based approach that enables autonomous vehicles to adaptively learn driving strategies without relying on predefined rules.
• To improve decision-making robustness and adaptability through reward function engineering in reinforcement learning.
• To compare the performance of DQN and PPO in real-world driving simulation environments.
• To validate the superiority of deep reinforcement learning over traditional rule-based systems in diverse driving tasks.

Proposed method

• Formulates the autonomous driving decision-making process as a Markov Decision Process (MDP) for reinforcement learning.
• Employs Deep Q-Network (DQN) to approximate the state-action value function and guide action selection.
• Applies Proximal Policy Optimization (PPO) to directly optimize the policy network for improved sample efficiency and training stability.
• Designs a customized reward function that incorporates safety, efficiency, and progress metrics to guide agent learning.
• Trains and evaluates the agents in a simulated environment with diverse traffic scenarios including intersections, merging, and pedestrian interactions.
• Uses curriculum learning and experience replay to enhance training convergence and generalization.

Experimental results

Research questions

• RQ1 Can deep reinforcement learning effectively learn robust and generalizable driving decision-making strategies in complex traffic environments?
• RQ2 How do DQN and PPO compare in performance and training stability for autonomous driving decision-making tasks?
• RQ3 To what extent does reward function design influence the adaptability and safety of learned driving policies?
• RQ4 Does the DRL-based approach outperform traditional rule-based systems in diverse driving scenarios?
• RQ5 How well do the learned policies generalize across unseen traffic situations?

Key findings

• The DRL-based decision-making system achieved significantly higher success rates across all evaluated driving tasks compared to rule-based baselines.
• PPO demonstrated superior training stability and faster convergence compared to DQN in complex traffic scenarios.
• The enhanced reward function improved policy generalization, reducing collision rates by 40% in high-density traffic environments.
• The model successfully handled dynamic interactions such as sudden lane changes and pedestrian crossings without explicit programming.
• The DRL agents learned to balance safety, efficiency, and comfort, achieving driving behaviors closer to human-like decision-making.
• The ablation study confirmed that reward shaping was critical for encouraging safe and efficient driving behaviors.