[Paper Review] Safety-Oriented Pedestrian Motion and Scene Occupancy Forecasting

Katie Luo, Sergio Casas+5 more|arXiv (Cornell University)|Jan 7, 2021

Autonomous Vehicle Technology and Safety65 references4 citations

TL;DR

This paper proposes a Scene-Actor Graph Neural Network (SA-GNN) that jointly forecasts pedestrian motions and scene-level occupancy maps to improve safety in autonomous driving. By modeling interactions among detected pedestrians and incorporating scene-wide occupancy predictions, SA-GNN achieves state-of-the-art performance in scene occupancy forecasting—especially in recall and calibration—while maintaining strong motion forecasting performance, even when pedestrians are missed by detection postprocessing.

ABSTRACT

In this paper, we address the important problem in self-driving of forecasting multi-pedestrian motion and their shared scene occupancy map, critical for safe navigation. Our contributions are two-fold. First, we advocate for predicting both the individual motions as well as the scene occupancy map in order to effectively deal with missing detections caused by postprocessing, e.g., confidence thresholding and non-maximum suppression. Second, we propose a Scene-Actor Graph Neural Network (SA-GNN) which preserves the relative spatial information of pedestrians via 2D convolution, and captures the interactions among pedestrians within the same scene, including those that have not been detected, via message passing. On two large-scale real-world datasets, nuScenes and ATG4D, we showcase that our scene-occupancy predictions are more accurate and better calibrated than those from state-of-the-art motion forecasting methods, while also matching their performance in pedestrian motion forecasting metrics.

Motivation & Objective

• To address safety-critical gaps in pedestrian perception and motion forecasting caused by detection failures such as confidence thresholding and non-maximum suppression.
• To improve scene occupancy prediction accuracy and calibration, especially for rare or low-probability events like undetected jaywalkers.
• To develop a unified framework that jointly predicts individual pedestrian trajectories and scene-level occupancy, enhancing robustness to missing detections.
• To demonstrate that scene-occupancy prediction is essential for high-recall, safety-oriented motion planning in real-world autonomous driving scenarios.

Proposed method

• The SA-GNN uses 2D convolutional features to preserve spatial relationships within each pedestrian's region of interest (RoI), reducing translation invariance issues.
• It introduces CoordConv to encode spatial coordinates in RoI features, improving localization accuracy for individual actors.
• The model constructs a graph with nodes for detected pedestrians and a dedicated scene node, enabling message passing between actors and the scene.
• It performs both actor-to-scene and scene-to-actor message passing, allowing the scene to inform pedestrian motion and vice versa.
• The network is trained end-to-end using a multi-task loss combining instance-level motion forecasting and scene occupancy prediction.
• The scene occupancy map is predicted in an instance-free, dense manner, avoiding reliance on postprocessed detections.

Experimental results

Research questions

• RQ1 Can joint prediction of pedestrian motion and scene occupancy improve safety in autonomous driving by compensating for detection failures?
• RQ2 How does modeling scene-level occupancy alongside individual pedestrian interactions affect prediction accuracy and calibration?
• RQ3 To what extent does including scene-to-actor message passing enhance robustness to undetected pedestrians?
• RQ4 Does a unified scene-occupancy and motion forecasting framework outperform separate, instance-based motion forecasting methods in terms of recall and reliability?
• RQ5 Can a model trained end-to-end achieve better calibration and generalization on real-world datasets like nuScenes and ATG4D?

Key findings

• On the ATG4D dataset, SA-GNN achieves a final mAP of 19.28 for scene occupancy, significantly outperforming the baseline FCN (15.71) and FCN+CoordConv (16.31).
• The model achieves a reliability diagram near perfect on ATG4D, indicating well-calibrated uncertainty estimates without postprocessing.
• In the ablation study, adding scene-to-actor messages reduced the average occupancy probability on ground-truth trajectories of undetected pedestrians from 9.25% to 5.46%, demonstrating improved robustness.
• SA-GNN achieves state-of-the-art performance in scene occupancy forecasting while matching or exceeding previous methods in instance-level motion forecasting metrics such as ACE (1.21), MCE (2.55), and NLL (4.02).
• The model maintains high precision at low recall regimes and exhibits a longer tail in recall, indicating better detection of rare or low-confidence pedestrians.
• The inclusion of scene-to-actor messages improves both scene occupancy and motion forecasting performance, showing that scene-level awareness enhances overall robustness.