[Paper Review] GPT-ST: Generative Pre-Training of Spatio-Temporal Graph Neural Networks

Zhonghang Li, Lianghao Xia+2 more|arXiv (Cornell University)|Nov 7, 2023

Traffic Prediction and Management TechniquesEngineering14 citations

TL;DR

GPT-ST introduces a spatio-temporal pre-training framework that uses masked autoencoding with a hierarchical hypergraph and adaptive masking to improve downstream spatio-temporal prediction models.

ABSTRACT

In recent years, there has been a rapid development of spatio-temporal prediction techniques in response to the increasing demands of traffic management and travel planning. While advanced end-to-end models have achieved notable success in improving predictive performance, their integration and expansion pose significant challenges. This work aims to address these challenges by introducing a spatio-temporal pre-training framework that seamlessly integrates with downstream baselines and enhances their performance. The framework is built upon two key designs: (i) We propose a spatio-temporal mask autoencoder as a pre-training model for learning spatio-temporal dependencies. The model incorporates customized parameter learners and hierarchical spatial pattern encoding networks. These modules are specifically designed to capture spatio-temporal customized representations and intra- and inter-cluster region semantic relationships, which have often been neglected in existing approaches. (ii) We introduce an adaptive mask strategy as part of the pre-training mechanism. This strategy guides the mask autoencoder in learning robust spatio-temporal representations and facilitates the modeling of different relationships, ranging from intra-cluster to inter-cluster, in an easy-to-hard training manner. Extensive experiments conducted on representative benchmarks demonstrate the effectiveness of our proposed method. We have made our model implementation publicly available at https://github.com/HKUDS/GPT-ST.

Motivation & Objective

• Motivate robust spatio-temporal representation learning for downstream prediction tasks.
• Enable customization of temporal and spatial representations to capture intra- and inter-cluster relations.
• Model global region similarities via hierarchical hypergraphs and capsule-based clustering.
• Facilitate end-to-end improvements by integrating pre-training with diverse ST baselines.

Proposed method

• Masked autoencoding pre-training to reconstruct masked spatio-temporal data via a ST representation function f.
• Customized temporal pattern encoding with time- and region-specific parameters generated by a parameter learner.
• Hierarchical spatial encoding using a hypergraph capsule clustering network to capture global region similarities.
• Cross-cluster relation learning with a high-level hypergraph to model inter-cluster dynamics.
• Cluster-aware masking strategy that progressively masks clusters to promote intra- and inter-cluster learning; KL-divergence loss aligns learned cluster signals with masking predictions.

Experimental results

Research questions

• RQ1 RQ1: How does GPT-ST improve downstream spatio-temporal prediction baselines across datasets?
• RQ2 RQ2: How effective are the designed modules (temporal encoding, hierarchical spatial encoding, masking) in GPT-ST?
• RQ3 RQ3: Can the learned global region clusters and inter-cluster transitions be interpreted meaningfully?
• RQ4 RQ4: What is the efficiency of GPT-ST in pre-training and downstream tasks?
• RQ5 RQ5: How does varying the mask ratio affect GPT-ST performance?

Key findings

• GPT-ST consistently improves downstream baselines across multiple ST datasets and models.
• The hypergraph capsule clustering and cross-cluster learning contribute notably to performance gains, with ablations showing the biggest drop when these components are removed.
• Adaptive cluster-aware masking outperforms random masking and other masking strategies in promoting intra- and inter-cluster representations.
• GPT-ST’s pre-training strategy yields improvements even when compared to STEP and long-term pre-training baselines, especially when long-term data is limited.
• Most gains are observed on classical baselines (e.g., STGCN) rather than highly engineered SOTA models, indicating GPT-ST complements simpler architectures.