[Paper Review] Hybrid Reinforcement Learning-Based Eco-Driving Strategy for Connected and Automated Vehicles at Signalized Intersections

Zhengwei Bai, Hao Peng+3 more|arXiv (Cornell University)|Jan 19, 2022

Traffic control and management4 citations

TL;DR

This paper proposes a hybrid reinforcement learning (HRL) framework that integrates rule-based strategies with deep reinforcement learning to enable eco-driving for connected and automated vehicles (CAVs) at signalized intersections in mixed traffic. By fusing vision-perceptive data and V2I communications, the HRL model optimizes both longitudinal and lateral driving actions, reducing energy consumption by 12.70% and travel time by 11.75% compared to a state-of-the-art model-based approach in Unity-based simulations.

ABSTRACT

Taking advantage of both vehicle-to-everything (V2X) communication and automated driving technology, connected and automated vehicles are quickly becoming one of the transformative solutions to many transportation problems. However, in a mixed traffic environment at signalized intersections, it is still a challenging task to improve overall throughput and energy efficiency considering the complexity and uncertainty in the traffic system. In this study, we proposed a hybrid reinforcement learning (HRL) framework which combines the rule-based strategy and the deep reinforcement learning (deep RL) to support connected eco-driving at signalized intersections in mixed traffic. Vision-perceptive methods are integrated with vehicle-to-infrastructure (V2I) communications to achieve higher mobility and energy efficiency in mixed connected traffic. The HRL framework has three components: a rule-based driving manager that operates the collaboration between the rule-based policies and the RL policy; a multi-stream neural network that extracts the hidden features of vision and V2I information; and a deep RL-based policy network that generate both longitudinal and lateral eco-driving actions. In order to evaluate our approach, we developed a Unity-based simulator and designed a mixed-traffic intersection scenario. Moreover, several baselines were implemented to compare with our new design, and numerical experiments were conducted to test the performance of the HRL model. The experiments show that our HRL method can reduce energy consumption by 12.70% and save 11.75% travel time when compared with a state-of-the-art model-based Eco-Driving approach.

Motivation & Objective

• To address the challenge of improving mobility and energy efficiency in mixed traffic at signalized intersections where traditional rule-based or model-based methods fail due to unrealistic assumptions.
• To develop a hybrid reinforcement learning framework that combines rule-based safety and efficiency with deep RL for adaptive, real-time eco-driving in complex traffic environments.
• To leverage V2I communications and on-board sensing (camera, radar) to enhance situational awareness and enable coordinated, energy-efficient vehicle trajectories.
• To evaluate the HRL framework in a realistic simulation environment under varying traffic penetration rates and signal timing conditions.
• To demonstrate superior performance over existing model-based and graph-based eco-driving methods in terms of energy savings and travel time reduction.

Proposed method

• The HRL framework consists of a rule-based driving manager that coordinates between rule-based policies and a deep RL policy to ensure safety and efficiency.
• A multi-stream neural network processes spatiotemporal data from front camera, on-board radar, OBD, and V2I signals to extract hidden features for decision-making.
• A deep RL-based policy network generates both longitudinal acceleration and lateral lane-target actions using a Long-Short Term Reward (LSTR) model that balances immediate and long-term benefits.
• The system is trained and evaluated in a Unity-based simulator with a mixed-traffic intersection scenario, including connected and automated vehicles (CAVs) and human-driven vehicles (HVs).
• The framework uses signal phase and timing (SPaT) and geometric intersection description (GID) data from V2I to anticipate signal changes and optimize approach speed.
• Performance is benchmarked against a graph-based model and the Intelligent Driver Model (IDM), with ablation studies including regenerative braking to validate robustness.

Experimental results

Research questions

• RQ1 Can a hybrid reinforcement learning framework effectively combine rule-based safety with deep RL for eco-driving in mixed traffic at signalized intersections?
• RQ2 How does the integration of V2I and on-board vision data improve energy efficiency and travel time compared to purely model-based or rule-based methods?
• RQ3 What is the impact of varying signal timing and vehicle penetration rates on the performance of the HRL-based eco-driving strategy?
• RQ4 Does the HRL model achieve better energy and time efficiency than state-of-the-art model-based approaches under realistic mixed-traffic conditions?
• RQ5 Where is the optimal operating region for the HRL model in terms of entry speed and time, and how does it affect performance?

Key findings

• The HRL method reduces average energy consumption by 12.70% and travel time by 11.75% compared to a state-of-the-art graph-based model across all 21 test scenarios.
• In the most favorable condition (entry speed 30 km/h, signal phase C50), the HRL model achieves a 42.94% reduction in energy consumption and 55.83% reduction in travel time compared to the graph-based method.
• With regenerative braking enabled, the HRL method reduces energy use by 23.60% and 41.59% compared to the graph-based model and IDM, respectively, confirming its robustness and scalability.
• The HRL framework outperforms the IDM in 21 out of 21 scenarios for energy consumption and in 27 out of 27 scenarios for travel time, demonstrating consistent superiority.
• Heat maps of performance improvements show that the HRL model performs best when vehicles enter the intersection at a moderate speed and optimal time, indicating a distinct performance sweet spot.
• The framework maintains consistent performance gains across varying signal phases and traffic conditions, validating its adaptability in dynamic mixed-traffic environments.