[Paper Review] Spatial-Temporal Transformer Networks for Traffic Flow Forecasting

Mingxing Xu, Wenrui Dai+5 more|arXiv (Cornell University)|Jan 9, 2020

Traffic Prediction and Management Techniques31 references265 citations

TL;DR

The paper introduces Spatial-Temporal Transformer Networks (STTNs) that model dynamic directed spatial dependencies and long-range temporal dependencies to improve long-term traffic flow forecasting. STTNs achieve competitive, state-of-the-art performance on PeMS-BAY and PeMSD7(M), especially for longer horizons.

ABSTRACT

Traffic forecasting has emerged as a core component of intelligent transportation systems. However, timely accurate traffic forecasting, especially long-term forecasting, still remains an open challenge due to the highly nonlinear and dynamic spatial-temporal dependencies of traffic flows. In this paper, we propose a novel paradigm of Spatial-Temporal Transformer Networks (STTNs) that leverages dynamical directed spatial dependencies and long-range temporal dependencies to improve the accuracy of long-term traffic forecasting. Specifically, we present a new variant of graph neural networks, named spatial transformer, by dynamically modeling directed spatial dependencies with self-attention mechanism to capture realtime traffic conditions as well as the directionality of traffic flows. Furthermore, different spatial dependency patterns can be jointly modeled with multi-heads attention mechanism to consider diverse relationships related to different factors (e.g. similarity, connectivity and covariance). On the other hand, the temporal transformer is utilized to model long-range bidirectional temporal dependencies across multiple time steps. Finally, they are composed as a block to jointly model the spatial-temporal dependencies for accurate traffic prediction. Compared to existing works, the proposed model enables fast and scalable training over a long range spatial-temporal dependencies. Experiment results demonstrate that the proposed model achieves competitive results compared with the state-of-the-arts, especially forecasting long-term traffic flows on real-world PeMS-Bay and PeMSD7(M) datasets.

Motivation & Objective

• Address the challenge of accurate long-term traffic forecasting amid highly dynamic spatial-temporal dependencies.
• Introduce a spatial transformer to capture time-varying directed spatial relationships.
• Incorporate a temporal transformer to model long-range temporal dependencies for multi-step prediction.
• Enable efficient, scalable training for joint spatial-temporal modeling on traffic networks.

Proposed method

• Define a spatial-temporal block architecture with a spatial transformer and a temporal transformer.
• Model dynamical spatial dependencies via a dynamical graph convolutional layer and a fixed graph convolutional layer with a gate fusion mechanism.
• Use learnable spatial-temporal positional embeddings to incorporate topology and time.
• Apply self-attention in the spatial and temporal transformers to capture long-range dependencies.
• Train using a two-layer prediction head to generate multi-step forecasts in parallel.
• Formulate STTN as a dynamical graph neural network with message-passing semantics.

Experimental results

Research questions

• RQ1 How can dynamic directed spatial dependencies be effectively modeled for traffic networks?
• RQ2 Can a temporal transformer capture long-range temporal dependencies to improve multi-step traffic forecasts?
• RQ3 Does integrating spatial and temporal transformers yield better long-term accuracy than fixed-spatial-layout models?

Key findings

• STTN delivers competitive results to state-of-the-art methods, with notable gains for long-term predictions on real-world datasets PeMSD7(M) and PEMS-BAY.
• The spatial transformer learns both stationary topology-driven and dynamical traffic-condition-driven spatial patterns via fixed and dynamical graph convolutions.
• The temporal transformer enables simultaneous multi-step prediction by modeling long-range temporal dependencies with self-attention.
• The gate mechanism effectively fuses spatial features from fixed and dynamical convolutions, improving robust feature representation.
• The model supports efficient parallel training and scalable long-range dependency modeling, exceeding prior approaches in long-term forecasting.