[Paper Review] Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction

Cunjun Yu, Xiao Ma+3 more|arXiv (Cornell University)|May 18, 2020

Anomaly Detection Techniques and ApplicationsComputer Science51 references35 citations

TL;DR

STAR uses interleaved spatial and temporal Transformers with TGConv graph convolution and an external memory to predict pedestrian trajectories with state-of-the-art performance on five datasets using only attention mechanisms.

ABSTRACT

Understanding crowd motion dynamics is critical to real-world applications, e.g., surveillance systems and autonomous driving. This is challenging because it requires effectively modeling the socially aware crowd spatial interaction and complex temporal dependencies. We believe attention is the most important factor for trajectory prediction. In this paper, we present STAR, a Spatio-Temporal grAph tRansformer framework, which tackles trajectory prediction by only attention mechanisms. STAR models intra-graph crowd interaction by TGConv, a novel Transformer-based graph convolution mechanism. The inter-graph temporal dependencies are modeled by separate temporal Transformers. STAR captures complex spatio-temporal interactions by interleaving between spatial and temporal Transformers. To calibrate the temporal prediction for the long-lasting effect of disappeared pedestrians, we introduce a read-writable external memory module, consistently being updated by the temporal Transformer. We show that with only attention mechanism, STAR achieves state-of-the-art performance on 5 commonly used real-world pedestrian prediction datasets.

Motivation & Objective

• Motivate accurate pedestrian trajectory prediction in crowded scenes.
• Model social interactions and temporal dependencies with attention-based mechanisms.
• Propose a Transformer-based graph convolution (TGConv) for spatial modeling.
• Capture spatio-temporal dynamics by interleaving spatial and temporal Transformers.
• Improve temporal consistency with a read-writable external graph memory.

Proposed method

• Introduce TGConv, a Transformer-based graph convolution to model spatial interactions.
• Apply a temporal Transformer to learn temporal dependencies for each pedestrian.
• Interleave spatial and temporal Transformers to capture coupled spatio-temporal dynamics.
• Add a read-writable external memory to smooth temporal embeddings across time steps.
• Use two encoder blocks and a simple decoder to predict future trajectories.
• Train end-to-end using Adam with predefined hyperparameters and evaluation on ADE/FDE metrics.

Experimental results

Research questions

• RQ1 Can attention-based STAR models outperform state-of-the-art social trajectory predictors on standard datasets?
• RQ2 Does interleaving spatial and temporal Transformers yield better spatio-temporal modeling than separate processing?
• RQ3 Does TGConv provide a superior spatial interaction modeling compared to traditional graph convolutions?
• RQ4 Does an external graph memory improve temporal consistency and prediction accuracy?

Key findings

• STAR-D (deterministic) outperforms solid baselines on multiple datasets; STAR (stochastic) achieves state-of-the-artPerformance with stochastic sampling.
• TGConv (Transformer-based graph convolution) delivers better spatial interaction modeling than GCN/GAT alternatives, especially in denser crowds.
• Interleaving two encoders (spatial then temporal, and vice versa) generally yields better spatio-temporal representations than a single encoder.
• The temporal Transformer improves over LSTM-based temporal modeling in trajectory prediction.
• External graph memory provides smoother temporal embeddings and improves overall performance, particularly in certain datasets.