[Paper Review] Formulation and validation of a car-following model based on deep reinforcement learning

Fabian Hart, Ostap Okhrin+1 more|arXiv (Cornell University)|Sep 29, 2021

Traffic control and managementEngineering33 references20 citations

TL;DR

This paper proposes a deep reinforcement learning (DRL)-based car-following model trained to maximize safety, comfort, and efficiency through customizable reward functions, using synthetic leader trajectories from a truncated Ornstein-Uhlenbeck process. The model achieves unconditional string stability, crash-free behavior, and superior performance over the IDM across diverse real and artificial leader data.

ABSTRACT

We propose and validate a novel car following model based on deep reinforcement learning. Our model is trained to maximize externally given reward functions for the free and car-following regimes rather than reproducing existing follower trajectories. The parameters of these reward functions such as desired speed, time gap, or accelerations resemble that of traditional models such as the Intelligent Driver Model (IDM) and allow for explicitly implementing different driving styles. Moreover, they partially lift the black-box nature of conventional neural network models. The model is trained on leading speed profiles governed by a truncated Ornstein-Uhlenbeck process reflecting a realistic leader's kinematics. This allows for arbitrary driving situations and an infinite supply of training data. For various parameterizations of the reward functions, and for a wide variety of artificial and real leader data, the model turned out to be unconditionally string stable, comfortable, and crash-free. String stability has been tested with a platoon of five followers following an artificial and a real leading trajectory. A cross-comparison with the IDM calibrated to the goodness-of-fit of the relative gaps showed a higher reward compared to the traditional model and a better goodness-of-fit.

Motivation & Objective

• To develop a car-following model that generalizes across diverse traffic scenarios, including free driving and safety-critical situations.
• To overcome limitations of existing DRL models, such as restricted acceleration ranges and lack of generalization beyond training data.
• To ensure string stability and crash-free behavior even in extreme braking scenarios.
• To enable explicit modeling of different driving styles through adjustable reward function parameters.
• To validate the model on both synthetic and real-world trajectory data, including platoons and real-world leader-follower experiments.

Proposed method

• The model uses a modularized DRL framework with two separate policies: one for free driving and one for car-following, trained via Deep Deterministic Policy Gradient (DDPG).
• Reward functions are designed to reflect safety (time-to-collision), comfort (acceleration smoothness), and desired speed, with parameters adjustable to emulate different driving styles.
• Leader vehicle trajectories are generated using a truncated Ornstein-Uhlenbeck process to simulate realistic kinematics and enable infinite, diverse training data.
• The model is trained end-to-end to maximize cumulative reward, with policy networks mapping state observations (relative speed, gap, leader speed) to acceleration actions.
• String stability is evaluated via platoon simulations with five followers, using both artificial and real leader trajectories.
• Performance is benchmarked against the IDM calibrated to match the same real trajectory data for goodness-of-fit and safety metrics.

Experimental results

Research questions

• RQ1 Can a DRL-based car-following model achieve string stability across diverse and extreme traffic scenarios, including full-braking maneuvers?
• RQ2 Does the model generalize beyond training data, particularly when tested on real-world leader trajectories?
• RQ3 How does the model’s performance in safety, comfort, and efficiency compare to the traditional IDM under identical conditions?
• RQ4 Can the model smoothly transition between free driving and car-following regimes without abrupt behavior?
• RQ5 To what extent can driver characteristics (e.g., aggressive vs. defensive) be explicitly modeled through reward function parameters?

Key findings

• The model achieved unconditional string stability in platoon simulations with both artificial and real leader trajectories, even under large acceleration outliers.
• The model was crash-free in all tested scenarios, including extreme braking events where the leader decelerated at approximately 6 m/s².
• The RL agent achieved a lower minimum time-to-collision (TTC) of 1.99 s compared to the IDM’s 2.04 s, with higher TTC values across most of the distribution.
• The model outperformed the IDM in cumulative reward by 1.9% on average across 15 real leader-follower experiments, indicating better alignment with safety and comfort objectives.
• The model demonstrated smooth, comfortable transitions between free driving and car-following states, with no abrupt changes in acceleration.
• The model generalized well beyond training data, maintaining safety and stability on real-world trajectories not seen during training.