[Paper Review] LargeST: A Benchmark Dataset for Large-Scale Traffic Forecasting

Xu Liu, Yutong Xia+8 more|arXiv (Cornell University)|Jun 14, 2023

Traffic Prediction and Management TechniquesEngineering20 citations

TL;DR

The paper presents LargeST, a large-scale traffic forecasting benchmark with 8,600 California sensors over 5 years (2017–2021), rich metadata, and a comprehensive baseline evaluation to study scalability, efficiency, and temporal distribution shifts.

ABSTRACT

Road traffic forecasting plays a critical role in smart city initiatives and has experienced significant advancements thanks to the power of deep learning in capturing non-linear patterns of traffic data. However, the promising results achieved on current public datasets may not be applicable to practical scenarios due to limitations within these datasets. First, the limited sizes of them may not reflect the real-world scale of traffic networks. Second, the temporal coverage of these datasets is typically short, posing hurdles in studying long-term patterns and acquiring sufficient samples for training deep models. Third, these datasets often lack adequate metadata for sensors, which compromises the reliability and interpretability of the data. To mitigate these limitations, we introduce the LargeST benchmark dataset. It encompasses a total number of 8,600 sensors in California with a 5-year time coverage and includes comprehensive metadata. Using LargeST, we perform in-depth data analysis to extract data insights, benchmark well-known baselines in terms of their performance and efficiency, and identify challenges as well as opportunities for future research. We release the datasets and baseline implementations at: https://github.com/liuxu77/LargeST.

Motivation & Objective

• Motivate the need for large-scale, long-horizon traffic forecasting benchmarks that reflect real-world networks.
• Introduce LargeST, a benchmark with 8,600 sensors across California, 5 years of data, and rich per-sensor metadata.
• Provide data analysis to reveal spatial, temporal, and metadata factors affecting forecasting.
• Benchmark a suite of baselines to assess accuracy, efficiency, and scalability on LargeST, and discuss challenges and future opportunities.

Proposed method

• Construct LargeST from CalTrans PeMS data, selecting mainline sensors, and generating a graph with adjacency based on road-network distances within a 4-km radius.
• Aggregate 5-minute readings to 15-minute windows to obtain 96 time steps per day across 5 years (2017–2021).
• Represent sensors with comprehensive metadata (coordinates, county, highway, lanes, etc.) to improve interpretability and model design.
• Build four sub-datasets (CA, GLA, GBA, SD) to enable scalable evaluation from medium to large graph sizes.
• Construct train/validation/test splits chronologically with a 6:2:2 ratio for all sub-datasets; use MAE, RMSE, and MAPE as core forecasting metrics, plus training and inference time as efficiency metrics.
• Evaluate a suite of baselines (HL, LSTM, DCRNN, AGCRN, STGCN, GWNET, ASTGCN, STTN, STGODE, DSTAGNN, DGCRN, D2STGNN) to analyze performance, efficiency, and scalability on LargeST.

Experimental results

Research questions

• RQ1 How does forecasting accuracy vary across large-scale sensor networks with five years of data?
• RQ2 What is the impact of incorporating sensor metadata on predictive performance and interpretability?
• RQ3 How do state-of-the-art baselines scale in terms of accuracy and efficiency on LargeST compared to smaller public datasets?
• RQ4 What challenges (e.g., temporal distribution shifts, model complexity) arise when scaling traffic forecasting models to large networks?
• RQ5 Can LargeST serve as a foundation for future scalable forecasting methods and potential foundation models in time series?

Key findings

• LargeST comprises 8,600 California sensors with 5 years of data, yielding 525,888 time frames.
• Baseline methods show varying performance, with GWNET and AGCRN delivering strong results across several sub-datasets, while some recent models struggle to scale to CA and GLA sizes.
• STGCN and STGODE can outperform GWNET on CA in some metrics but require more parameters, highlighting a trade-off between accuracy and model size.
• Dynamic adjacency and metadata-aware models (e.g., DGCRN, D2 STGNN) perform well on smaller sub-datasets (SD, GBA) but face scalability challenges on larger graphs like CA and GLA.
• CA dataset presents scalability challenges where many baselines fail due to memory constraints, underscoring the need for scalable traffic forecasting approaches.
• The dataset supports analysis of temporal distribution shifts (including pandemic years 2020–2021) and encourages development of simple yet effective models and foundation forecasting approaches.