[Paper Review] Graph-SIM: A Graph-based Spatiotemporal Interaction Modelling for Pedestrian Action Prediction

Tiffany Yau, Saber Malekmohammadi+4 more|arXiv (Cornell University)|Dec 3, 2020

Autonomous Vehicle Technology and Safety37 references4 citations

TL;DR

This paper proposes Graph-SIM, a graph-based spatiotemporal interaction model that predicts pedestrian crossing actions by modeling dynamic interactions using bird's-eye-view 3D coordinates, orientations, and relative importance weights. It introduces a new dataset, PePScenes, with 3D bounding boxes and behavioral annotations, achieving state-of-the-art performance with over 15% improvement across key metrics compared to existing methods.

ABSTRACT

One of the most crucial yet challenging tasks for autonomous vehicles in urban environments is predicting the future behaviour of nearby pedestrians, especially at points of crossing. Predicting behaviour depends on many social and environmental factors, particularly interactions between road users. Capturing such interactions requires a global view of the scene and dynamics of the road users in three-dimensional space. This information, however, is missing from the current pedestrian behaviour benchmark datasets. Motivated by these challenges, we propose 1) a novel graph-based model for predicting pedestrian crossing action. Our method models pedestrians' interactions with nearby road users through clustering and relative importance weighting of interactions using features obtained from the bird's-eye-view. 2) We introduce a new dataset that provides 3D bounding box and pedestrian behavioural annotations for the existing nuScenes dataset. On the new data, our approach achieves state-of-the-art performance by improving on various metrics by more than 15% in comparison to existing methods. The dataset is available at https://github.com/huawei-noah/datasets/PePScenes.

Motivation & Objective

• To address the lack of 3D spatial and interaction-aware features in existing pedestrian behavior prediction benchmarks.
• To improve pedestrian crossing prediction by modeling dynamic interactions using bird's-eye-view 3D coordinates and motion directions.
• To develop a graph-based interaction modeling approach that weights interactions based on relative importance, incorporating distance, orientation, and traffic direction.
• To introduce a new dataset, PePScenes, extending nuScenes with 3D bounding boxes and pedestrian action annotations for improved benchmarking.
• To achieve state-of-the-art performance in pedestrian crossing prediction by integrating ego-vehicle dynamics and pedestrian motion features.

Proposed method

• The model constructs a pedestrian-centric star graph where nodes represent road users (pedestrians and vehicles) using their 3D global coordinates and orientations from a bird's-eye-view perspective.
• Edge weights are computed using a relative importance function that combines distance and directional alignment between road users, capturing traffic context more accurately than distance-only methods.
• The graph is processed through a graph neural network (GNN) to aggregate interaction features, which are fused with temporal dynamics from the target pedestrian and ego-vehicle using a recurrent architecture.
• Clustering is applied to group pedestrians based on proximity and orientation to model social forces and collective behavior, improving interaction representation.
• The final prediction head uses a multi-level fusion of dynamics features (location, velocity) and graph-encoded interactions to classify crossing probability.
• The model is trained and evaluated on a newly introduced dataset, PePScenes, which extends nuScenes with dense 3D bounding boxes and behavioral annotations for pedestrians.

Experimental results

Research questions

• RQ1 How can spatiotemporal interactions between pedestrians and surrounding road users be effectively modeled using 3D spatial and directional information?
• RQ2 To what extent does incorporating ego-vehicle dynamics improve pedestrian crossing prediction performance?
• RQ3 Can clustering based on 3D coordinates and orientations better capture social group behavior than proximity-based grouping alone?
• RQ4 How does relative importance weighting—based on distance and motion direction—improve interaction modeling compared to uniform or distance-only weighting?
• RQ5 Can a graph-based model with 3D bird's-eye-view features outperform existing SOTA methods on pedestrian action prediction?

Key findings

• The complete Graph-SIM model, incorporating both pedestrian and ego-vehicle dynamics, achieved 94.4% accuracy, 85.8% AUC, 81.4% F1, and 92.1% precision on the PePScenes dataset.
• The model improved F1 and precision by 29% and 20% respectively over the baseline Social-STGCNN, demonstrating significant performance gains.
• Including ego-vehicle dynamics improved performance more than pedestrian dynamics alone, highlighting its dominant influence on pedestrian behavior.
• Relative importance modeling with directional awareness outperformed distance-only weighting, improving F1 by 7.6 percentage points over Social-STGCNN.
• Clustering based on orientation and proximity improved precision by 5% compared to clustering using proximity alone, indicating better group behavior modeling.
• The ablation study confirmed that combining pedestrian location and velocity features yields significantly better performance than using them in isolation.