[Paper Review] EnsembleFollower: A Hybrid Car-Following Framework Based On Reinforcement Learning and Hierarchical Planning

Xu Han, Xianda Chen+4 more|arXiv (Cornell University)|Aug 30, 2023

Autonomous Vehicle Technology and Safety4 citations

TL;DR

This paper proposes EnsembleFollower, a hierarchical reinforcement learning framework that dynamically selects or blends multiple low-level car-following models (e.g., IDM, Gipps, FVD, RNN, DDPG) based on driving context to improve accuracy and robustness in autonomous vehicle longitudinal control. Evaluated on the HighD dataset, it outperforms individual models and state-of-the-art methods by leveraging complementary strengths through RL-based decision-making.

ABSTRACT

Car-following models have made significant contributions to our understanding of longitudinal driving behavior. However, they often exhibit limited accuracy and flexibility, as they cannot fully capture the complexity inherent in car-following processes, or may falter in unseen scenarios due to their reliance on confined driving skills present in training data. It is worth noting that each car-following model possesses its own strengths and weaknesses depending on specific driving scenarios. Therefore, we propose EnsembleFollower, a hierarchical planning framework for achieving advanced human-like car-following. The EnsembleFollower framework involves a high-level Reinforcement Learning-based agent responsible for judiciously managing multiple low-level car-following models according to the current state, either by selecting an appropriate low-level model to perform an action or by allocating different weights across all low-level components. Moreover, we propose a jerk-constrained kinematic model for more convincing car-following simulations. We evaluate the proposed method based on real-world driving data from the HighD dataset. The experimental results illustrate that EnsembleFollower yields improved accuracy of human-like behavior and achieves effectiveness in combining hybrid models, demonstrating that our proposed framework can handle diverse car-following conditions by leveraging the strengths of various low-level models.

Motivation & Objective

• To address the limitations of single car-following models in capturing complex, diverse driving behaviors under varying conditions.
• To improve generalization and safety in autonomous vehicle longitudinal control by integrating multiple complementary models.
• To develop a hierarchical framework that intelligently selects or blends low-level models based on real-time driving states.
• To validate the framework’s effectiveness using real-world driving data from the HighD dataset.

Proposed method

• Employs a high-level reinforcement learning (RL) agent (using DDQN or PPO) to manage multiple low-level car-following models based on current driving state.
• Supports two coordination mechanisms: discrete model selection (via DDQN) and convex combination of model outputs (via PPO) with learned weights.
• Integrates a jerk-constrained kinematic model to ensure smooth, realistic vehicle motion during simulation.
• Trains the RL agent using interactions with a simulator based on real-world HighD trajectory data.
• Uses rule-based models (Gipps, IDM, FVD) and data-driven models (RNN, DDPG) as low-level components with calibrated parameters and hyperparameters.
• Applies curriculum learning and experience replay to stabilize training and improve sample efficiency.

Experimental results

Research questions

• RQ1 Can a hierarchical RL framework effectively combine multiple low-level car-following models to improve longitudinal driving behavior accuracy?
• RQ2 How does the performance of the ensemble framework compare to individual rule-based and data-driven models on real-world driving data?
• RQ3 Does the proposed framework generalize better to unseen or rare driving scenarios than single-model baselines?
• RQ4 Which coordination strategy—discrete selection or convex combination—yields superior performance and robustness in diverse driving conditions?

Key findings

• EnsembleFollower significantly improves accuracy in replicating human-like car-following behavior compared to individual rule-based and data-driven models on the HighD dataset.
• The convex combination strategy (EF-PPO) achieves better performance than discrete selection (EF-DDQN), particularly in handling complex and dynamic traffic scenarios.
• The IDM model was most frequently selected in the EF-DDQN framework, indicating its strong baseline robustness across diverse conditions.
• The Gipps model was prioritized in high-speed and high-deceleration scenarios, demonstrating its effectiveness in safety-critical situations.
• Minority models such as FVD and RNN contributed significantly in specialized cases, enhancing the ensemble’s generalization capability.
• The framework effectively balances safety and efficiency by dynamically allocating model weights based on real-time driving states, as shown in attention heatmaps of model usage.