[Paper Review] Eco-driving for Electric Connected Vehicles at Signalized Intersections: A Parameterized Reinforcement Learning approach

Xia Jiang, Jian Zhang+1 more|arXiv (Cornell University)|Jun 24, 2022

Traffic control and management43 references4 citations

TL;DR

This paper proposes a parameterized reinforcement learning (PRL) framework for eco-driving in electric connected vehicles at signalized intersections, integrating model-based car-following and lane-changing policies with a hierarchical action space to jointly optimize longitudinal and lateral control. The method reduces energy consumption by up to 27.13% in non-coordinate signal settings without disrupting human-driven vehicles (HDVs).

ABSTRACT

This paper proposes an eco-driving framework for electric connected vehicles (CVs) based on reinforcement learning (RL) to improve vehicle energy efficiency at signalized intersections. The vehicle agent is specified by integrating the model-based car-following policy, lane-changing policy, and the RL policy, to ensure safe operation of a CV. Subsequently, a Markov Decision Process (MDP) is formulated, which enables the vehicle to perform longitudinal control and lateral decisions, jointly optimizing the car-following and lane-changing behaviors of the CVs in the vicinity of intersections. Then, the hybrid action space is parameterized as a hierarchical structure and thereby trains the agents with two-dimensional motion patterns in a dynamic traffic environment. Finally, our proposed methods are evaluated in SUMO software from both a single-vehicle-based perspective and a flow-based perspective. The results show that our strategy can significantly reduce energy consumption by learning proper action schemes without any interruption of other human-driven vehicles (HDVs).

Motivation & Objective

• To improve energy efficiency of electric connected vehicles (CVs) at signalized intersections through intelligent control.
• To address the limitations of rule-based and optimization-based eco-driving strategies in dynamic traffic environments.
• To develop a learning-based control framework that jointly optimizes longitudinal and lateral vehicle decisions.
• To ensure minimal disruption to human-driven vehicles (HDVs) during CV deployment in mixed-traffic scenarios.
• To evaluate the scalability and robustness of the proposed method under varying market penetration rates (MPR) of CVs and traffic volumes.

Proposed method

• Formulates a Markov Decision Process (MDP) to model joint longitudinal and lateral control decisions at intersections.
• Introduces a hybrid action space parameterized as a hierarchical structure to represent two-dimensional motion patterns.
• Integrates model-based car-following and lane-changing policies with a parameterized deep reinforcement learning (PRL) agent.
• Employs a composite reward function combining step-wise and terminal rewards to guide policy learning.
• Trains the PRL agent using deep reinforcement learning to learn optimal speed profiles and lane-change decisions based on SPaT (Signal Phase and Timing) data.
• Validates the framework in SUMO simulations under both single-agent and multi-agent scenarios with coordinated and non-coordinated signal settings.

Experimental results

Research questions

• RQ1 Can a parameterized reinforcement learning framework effectively reduce energy consumption in electric connected vehicles at signalized intersections?
• RQ2 How does the proposed PRL-based method compare to DQN and DDPG in terms of energy efficiency and stability?
• RQ3 To what extent does the framework maintain traffic efficiency for human-driven vehicles (HDVs) when CVs are deployed?
• RQ4 How does the market penetration rate (MPR) of CVs affect the overall energy and mobility performance of the traffic system?
• RQ5 Does the method perform better in coordinated versus non-coordinated signal environments?

Key findings

• The PRL-based framework reduces average electricity consumption by 13.8% compared to a pure HDV scenario when CV MPR is 1.
• In non-coordinate signal settings, the method achieves a 27.13% reduction in energy consumption compared to baseline strategies.
• In coordinated signal settings, the method reduces energy use by 4.1% compared to baseline, demonstrating consistent performance across signal types.
• The average speed of HDVs remains stable across varying MPR levels and traffic volumes, indicating no significant disruption to human-driven traffic.
• The PRL controller outperforms DQN and DDPG in both energy efficiency and stability under dynamic traffic conditions.
• The framework maintains low speed reduction (minimal mobility sacrifice) even at high MPR levels, confirming its practicality for real-world deployment.