[Paper Review] PDFormer: Propagation Delay-Aware Dynamic Long-Range Transformer for Traffic Flow Prediction

Jiawei Jiang, Chengkai Han+2 more|arXiv (Cornell University)|Jan 19, 2023

Traffic Prediction and Management Techniques22 citations

TL;DR

PDFormer introduces propagation delay-aware spatial-temporal self-attention with short- and long-range masking to predict traffic flow, achieving state-of-the-art accuracy and interpretable attention maps.

ABSTRACT

As a core technology of Intelligent Transportation System, traffic flow prediction has a wide range of applications. The fundamental challenge in traffic flow prediction is to effectively model the complex spatial-temporal dependencies in traffic data. Spatial-temporal Graph Neural Network (GNN) models have emerged as one of the most promising methods to solve this problem. However, GNN-based models have three major limitations for traffic prediction: i) Most methods model spatial dependencies in a static manner, which limits the ability to learn dynamic urban traffic patterns; ii) Most methods only consider short-range spatial information and are unable to capture long-range spatial dependencies; iii) These methods ignore the fact that the propagation of traffic conditions between locations has a time delay in traffic systems. To this end, we propose a novel Propagation Delay-aware dynamic long-range transFormer, namely PDFormer, for accurate traffic flow prediction. Specifically, we design a spatial self-attention module to capture the dynamic spatial dependencies. Then, two graph masking matrices are introduced to highlight spatial dependencies from short- and long-range views. Moreover, a traffic delay-aware feature transformation module is proposed to empower PDFormer with the capability of explicitly modeling the time delay of spatial information propagation. Extensive experimental results on six real-world public traffic datasets show that our method can not only achieve state-of-the-art performance but also exhibit competitive computational efficiency. Moreover, we visualize the learned spatial-temporal attention map to make our model highly interpretable.

Motivation & Objective

• Motivate accurate traffic flow prediction by addressing dynamic, long-range, and time-delay spatial dependencies.
• Model dynamic spatial relationships with a spatial self-attention module that fuses local geographic and global semantic neighborhoods.
• Explicitly incorporate propagation delays of traffic information into the prediction process.
• Capture dynamic temporal patterns and provide an interpretable spatial-temporal attention mechanism.

Proposed method

• Proposes PDFormer built on spatial-temporal self-attention with three core components: GeoSSA and SemSSA for short-/long-range spatial dependencies, and TSA for temporal dynamics.
• Introduces a delay-aware feature transformation (DFT) that augments key representations with historical pattern memory using k-Shape clustering and DTW-based semantic neighbors.
• Uses two graph masking matrices M_geo and M_sem to emphasize local geographic and global semantic relationships in attention computations.
• Employs a multi-head heterogeneous attention fusion that combines GeoSSA, SemSSA, and TSA outputs before a feed-forward network and residual connections.
• Data embedding layer includes graph Laplacian embedding and temporal periodic embeddings to inject structural and periodic information.
• Output layer uses skip connections and 1x1 convolutions for multi-step prediction without recursive decoding.

Experimental results

Research questions

• RQ1 How can dynamic spatial dependencies be effectively modeled for traffic data?”,

Key findings

• PDFormer achieves state-of-the-art performance across six real-world traffic datasets (three graph-based and three grid-based) on MAE, MAPE, and RMSE.
• In ablations, SSA outperforms GCN due to its ability to capture dynamic and long-range spatial dependencies.
• The two masking matrices (M_geo and M_sem) significantly improve performance by guiding attention to relevant local and distant-but-similar regions.
• Delay-aware feature transformation improves prediction by explicitly modeling spatial propagation delays.
• PDFormer offers competitive training and inference efficiency, outperforming baselines in several efficiency metrics.
• Visualization of spatial-temporal attention demonstrates interpretability by showing focus on nearby and semantically similar regions.