[Paper Review] V2X-Seq: A Large-Scale Sequential Dataset for Vehicle-Infrastructure Cooperative Perception and Forecasting

Haibao Yu, Wenxian Yang+12 more|arXiv (Cornell University)|May 10, 2023

Autonomous Vehicle Technology and Safety4 citations

TL;DR

V2X-Seq introduces the first large-scale, real-world sequential dataset for vehicle-infrastructure cooperative (VIC) perception and forecasting, comprising 15,000 frames for perception and 210,000 scenarios for trajectory forecasting across 672 hours of data. The paper proposes three new tasks—VIC3D Tracking, Online-VIC Forecasting, and Offline-VIC Forecasting—along with benchmarks, and demonstrates that integrating infrastructure data improves tracking and forecasting performance, with FF-Tracking achieving up to 4% MOTA gain over early fusion at 300ms latency.

ABSTRACT

Utilizing infrastructure and vehicle-side information to track and forecast the behaviors of surrounding traffic participants can significantly improve decision-making and safety in autonomous driving. However, the lack of real-world sequential datasets limits research in this area. To address this issue, we introduce V2X-Seq, the first large-scale sequential V2X dataset, which includes data frames, trajectories, vector maps, and traffic lights captured from natural scenery. V2X-Seq comprises two parts: the sequential perception dataset, which includes more than 15,000 frames captured from 95 scenarios, and the trajectory forecasting dataset, which contains about 80,000 infrastructure-view scenarios, 80,000 vehicle-view scenarios, and 50,000 cooperative-view scenarios captured from 28 intersections' areas, covering 672 hours of data. Based on V2X-Seq, we introduce three new tasks for vehicle-infrastructure cooperative (VIC) autonomous driving: VIC3D Tracking, Online-VIC Forecasting, and Offline-VIC Forecasting. We also provide benchmarks for the introduced tasks. Find data, code, and more up-to-date information at \href{https://github.com/AIR-THU/DAIR-V2X-Seq}{https://github.com/AIR-THU/DAIR-V2X-Seq}.

Motivation & Objective

• Address the lack of real-world sequential datasets for vehicle-infrastructure cooperative (VIC) perception and forecasting in autonomous driving.
• Enable research on cooperative tracking and behavior forecasting by providing a large-scale, real-world dataset with synchronized multi-modal data.
• Introduce three novel tasks—VIC3D Tracking, Online-VIC Forecasting, and Offline-VIC Forecasting—to advance VIC-based autonomous driving systems.
• Establish fair benchmarks for evaluating VIC perception and forecasting models using real-world data from 28 intersections over 672 hours.
• Develop and validate a novel FF-Tracking framework that improves tracking robustness under variable latency conditions.

Proposed method

• Collect and annotate a large-scale sequential V2X dataset (V2X-Seq) from 95 real-world scenarios, including infrastructure and vehicle-side images, point clouds, 3D detections, vector maps, and traffic light signals.
• Design and implement three new tasks: VIC3D Tracking (cooperative 3D object tracking), Online-VIC Forecasting (real-time behavior prediction using live infrastructure and ego-vehicle data), and Offline-VIC Forecasting (knowledge distillation from historical infrastructure data).
• Propose FF-Tracking, a late-fusion-based framework that combines vehicle and infrastructure inputs at the feature level to improve tracking accuracy and latency robustness.
• Develop PP-VIC, a hierarchical perception-prediction framework that fuses infrastructure and ego-vehicle trajectories before feeding them into trajectory models like TNT and HiVT.
• Pre-train trajectory forecasting models (TNT, HiVT) on 80,000 infrastructure-view trajectories to extract generalizable knowledge for offline forecasting.
• Use a 5:2:3 train/val/test split on the trajectory forecasting dataset to ensure fair evaluation of all baselines and proposed methods.

Figure 1 : Autonomous driving datasets. V2X-Seq is the first large-scale, real-world, and sequential V2X dataset. The green circle denotes the real-world dataset, and the pink triangle denotes the simulated dataset. The abscissa represents the number of sequences.

Experimental results

Research questions

• RQ1 Can infrastructure data significantly improve the accuracy and robustness of 3D object tracking in cooperative perception systems?
• RQ2 To what extent does real-time access to infrastructure trajectories enhance online trajectory forecasting performance compared to ego-vehicle-only models?
• RQ3 Can knowledge extracted from historical infrastructure trajectories improve offline trajectory forecasting performance?
• RQ4 How does the proposed FF-Tracking framework compare to early fusion baselines in terms of tracking performance under varying latency conditions?
• RQ5 What is the impact of pre-training on infrastructure-only trajectories for improving downstream forecasting model generalization?

Key findings

• The FF-Tracking framework outperforms early fusion by up to 4% in MOTA at 300ms latency, demonstrating superior robustness to communication delays.
• At 200ms latency, FF-Tracking achieves the best tracking performance, confirming its effectiveness in real-time cooperative perception.
• Using online infrastructure trajectories reduces minADE by 3.74 (TNT) and 0.28 (HiVT) compared to ego-vehicle-only baselines, proving their value in real-time forecasting.
• Pre-training on 80,000 infrastructure-view trajectories reduces minADE by 7.65 for TNT and 0.03 for HiVT compared to models without infrastructure data, showing the benefit of offline knowledge extraction.
• The PP-VIC framework achieves lower minADE and minFDE than ego-vehicle-only models, confirming that fusing infrastructure and vehicle trajectories improves forecasting accuracy.
• The V2X-Seq dataset enables the first large-scale, real-world evaluation of VIC perception and forecasting, with 50,000 cooperative-view scenarios and 672 hours of data, setting a new benchmark for the field.

Figure 2 : Total number and average tracking length of 3D tracked objects per category for the sequential perception dataset (SPD). The distribution of tracked objects is relatively balanced.