[Paper Review] Learning dynamic and hierarchical traffic spatiotemporal features with Transformer

Haoyang Yan, Xiaolei Ma|arXiv (Cornell University)|Apr 12, 2021

Traffic Prediction and Management Techniques24 citations

TL;DR

The paper introduces Traffic Transformer to model dynamic hierarchical spatiotemporal traffic features for long-term forecasting using multi-head attention and masked attention, outperforming state-of-the-art methods.

ABSTRACT

Traffic forecasting is an indispensable part of Intelligent transportation systems (ITS), and long-term network-wide accurate traffic speed forecasting is one of the most challenging tasks. Recently, deep learning methods have become popular in this domain. As traffic data are physically associated with road networks, most proposed models treat it as a spatiotemporal graph modeling problem and use Graph Convolution Network (GCN) based methods. These GCN-based models highly depend on a predefined and fixed adjacent matrix to reflect the spatial dependency. However, the predefined fixed adjacent matrix is limited in reflecting the actual dependence of traffic flow. This paper proposes a novel model, Traffic Transformer, for spatial-temporal graph modeling and long-term traffic forecasting to overcome these limitations. Transformer is the most popular framework in Natural Language Processing (NLP). And by adapting it to the spatiotemporal problem, Traffic Transformer hierarchically extracts spatiotemporal features through data dynamically by multi-head attention and masked multi-head attention mechanism, and fuse these features for traffic forecasting. Furthermore, analyzing the attention weight matrixes can find the influential part of road networks, allowing us to learn the traffic networks better. Experimental results on the public traffic network datasets and real-world traffic network datasets generated by ourselves demonstrate our proposed model achieves better performance than the state-of-the-art ones.

Motivation & Objective

• Motivate the need to move beyond fixed adjacency in GCN-based traffic forecasting.
• Propose a Transformer-based framework to capture dynamic spatiotemporal dependencies in traffic networks.
• Extract hierarchical spatiotemporal features through data-driven attention mechanisms.
• Enable interpretation of network influence by analyzing attention weights.

Proposed method

• Adapt Transformer architecture to spatial-temporal graph modeling for traffic data.
• Use multi-head attention to learn dynamic spatiotemporal dependencies and masked multi-head attention for temporal forecasting.
• Hierarchically fuse spatiotemporal features extracted from attention layers for final forecasting.
• Leverage attention weight matrices to identify influential road network components.
• Demonstrate improved forecasting performance on public and real-world datasets.

Experimental results

Research questions

• RQ1 Can Traffic Transformer model dynamic and hierarchical spatiotemporal dependencies in traffic networks?
• RQ2 How do attention weight matrices reflect influential parts of the road network for forecasting?
• RQ3 Does the proposed model improve long-term traffic forecasting compared to state-of-the-art methods?

Key findings

• Traffic Transformer achieves better performance than state-of-the-art methods on public traffic datasets and real-world datasets.
• Attention mechanisms enable hierarchical extraction of spatiotemporal features from traffic data.
• Analysis of attention weights helps identify influential roads and network regions for traffic forecasting.