[Paper Review] FollowNet: A Comprehensive Benchmark for Car-Following Behavior Modeling

Xianda Chen, Meixin Zhu+6 more|arXiv (Cornell University)|May 25, 2023

Traffic Prediction and Management Techniques47 references4 citations

TL;DR

FollowNet introduces the first standardized benchmark for car-following behavior modeling, extracting over 80,000 events from five public datasets using consistent criteria. It evaluates state-of-the-art models using standardized metrics, showing that the DDPG-based model achieves lower spacing MSE than traditional models (IDM, GHR) and zero collision rate—surpassing neural networks and LSTMs in safety and accuracy.

ABSTRACT

Car-following is a control process in which a following vehicle (FV) adjusts its acceleration to keep a safe distance from the lead vehicle (LV). Recently, there has been a booming of data-driven models that enable more accurate modeling of car-following through real-world driving datasets. Although there are several public datasets available, their formats are not always consistent, making it challenging to determine the state-of-the-art models and how well a new model performs compared to existing ones. In contrast, research fields such as image recognition and object detection have benchmark datasets like ImageNet, Microsoft COCO, and KITTI. To address this gap and promote the development of microscopic traffic flow modeling, we establish a public benchmark dataset for car-following behavior modeling. The benchmark consists of more than 80K car-following events extracted from five public driving datasets using the same criteria. These events cover diverse situations including different road types, various weather conditions, and mixed traffic flows with autonomous vehicles. Moreover, to give an overview of current progress in car-following modeling, we implemented and tested representative baseline models with the benchmark. Results show that the deep deterministic policy gradient (DDPG) based model performs competitively with a lower MSE for spacing compared to traditional intelligent driver model (IDM) and Gazis-Herman-Rothery (GHR) models, and a smaller collision rate compared to fully connected neural network (NN) and long short-term memory (LSTM) models in most datasets. The established benchmark will provide researchers with consistent data formats and metrics for cross-comparing different car-following models, promoting the development of more accurate models. We open-source our dataset and implementation code in https://github.com/HKUST-DRIVE-AI-LAB/FollowNet.

Motivation & Objective

• Address the lack of standardized data formats and evaluation criteria in car-following research, which hinders fair comparison of new models.
• Create a unified benchmark that enables consistent evaluation and cross-comparison of car-following models across diverse driving scenarios.
• Enhance model development by including mixed traffic flows with autonomous vehicles, which are underrepresented in existing datasets.
• Provide open access to curated data and code to accelerate research and reduce redundant data extraction efforts.
• Evaluate the performance of representative baseline models using consistent metrics to establish a performance baseline for future work.

Proposed method

• Extracted over 80,000 car-following events from five public driving datasets (NGSIM, HighD, Lyft, Waymo, and others) using uniform criteria for event selection.
• Standardized data formats and evaluation metrics—specifically mean squared error (MSE) of spacing and collision rate—across all datasets for consistent model comparison.
• Implemented five baseline models: traditional (IDM, GHR) and data-driven (Fully Connected NN, LSTM, DDPG) to serve as performance reference points.
• Calibrated traditional models using genetic algorithm (GA) to optimize parameters for improved safety and performance.
• Trained and evaluated deep learning models (LSTM, NN, DDPG) on the same standardized data split across all datasets.
• Released the dataset and codebase via GitHub (https://github.com/HKUST-DRIVE-AI-LAB/FollowNet) for community reuse and extension.

Experimental results

Research questions

• RQ1 How do data-driven models (e.g., DDPG, LSTM, NN) compare to traditional models (IDM, GHR) in terms of spacing prediction accuracy and collision avoidance?
• RQ2 To what extent can deep reinforcement learning models like DDPG generalize across diverse road types, weather conditions, and traffic compositions?
• RQ3 What is the performance gap between models trained on one dataset and tested on another, indicating generalization capability?
• RQ4 Can a unified benchmark with consistent data formats and evaluation metrics significantly reduce the effort and inconsistency in car-following model evaluation?
• RQ5 How do models perform in mixed traffic scenarios involving autonomous and human-driven vehicles, and what improvements are needed for safety and robustness?

Key findings

• The DDPG-based model (DDPG_Max) achieved a lower mean squared error (MSE) of spacing than traditional models (IDM and GHR) across most datasets, indicating superior prediction accuracy.
• Traditional models (IDM, GHR) calibrated with genetic algorithm (GA) achieved a zero collision rate, highlighting the importance of safety in model design.
• Data-driven models such as fully connected neural networks (NN) and LSTMs exhibited higher collision rates compared to DDPG and traditional models, indicating safety trade-offs.
• The DDPG_Max model achieved both a lower MSE of spacing than IDM and GHR and a zero collision rate—surpassing both traditional and other data-driven models in combined performance.
• The benchmark revealed significant performance discrepancies when models trained on one dataset (e.g., NGSIM) were tested on another (e.g., Lyft), indicating limited generalization across datasets.
• The inclusion of autonomous vehicles in datasets like Lyft and Waymo is critical for modeling mixed traffic flows, and such data is essential for future model development.