[Paper Review] Scenario-Transferable Semantic Graph Reasoning for Interaction-Aware Probabilistic Prediction

Yeping Hu, Wei Zhan+1 more|arXiv (Cornell University)|Apr 7, 2020

Autonomous Vehicle Technology and SafetyEngineering59 references17 citations

TL;DR

This paper proposes a scenario-transferable semantic graph reasoning framework for interaction-aware probabilistic prediction in autonomous driving. By leveraging semantics and domain knowledge to create generic, domain-invariant environment representations, and modeling them as spatial-temporal semantic graphs, the method enables state-of-the-art prediction performance with strong zero-shot transferability across diverse driving scenarios without retraining.

ABSTRACT

Accurately predicting the possible behaviors of traffic participants is an essential capability for autonomous vehicles. Since autonomous vehicles need to navigate in dynamically changing environments, they are expected to make accurate predictions regardless of where they are and what driving circumstances they encountered. Several methodologies have been proposed to solve prediction problems under different traffic situations. These works usually combine agent trajectories with either color-coded or vectorized high definition (HD) map as input representations and encode this information for behavior prediction tasks. However, not all the information is relevant in the scene for the forecasting and such irrelevant information may be even distracting to the forecasting in certain situations. Therefore, in this paper, we propose a novel generic representation for various driving environments by taking the advantage of semantics and domain knowledge. Using semantics enables situations to be modeled in a uniform way and applying domain knowledge filters out unrelated elements to target vehicle's future behaviors. We then propose a general semantic behavior prediction framework to effectively utilize these representations by formulating them into spatial-temporal semantic graphs and reasoning internal relations among these graphs. We theoretically and empirically validate the proposed framework under highly interactive and complex scenarios, demonstrating that our method not only achieves state-of-the-art performance, but also processes desirable zero-shot transferability.

Motivation & Objective

• To develop a generic, scenario-agnostic representation of driving environments that captures relevant semantic and structural information while filtering out irrelevant elements using domain knowledge.
• To address the challenge of generalizing prediction models across diverse, unseen driving scenarios with different road structures, traffic rules, and agent interactions.
• To enable zero-shot transferability of trained prediction models to new environments without retraining or data collection.
• To improve prediction accuracy and confidence in complex, interactive urban scenarios by modeling spatial-temporal relations through semantic graphs.

Proposed method

• The method introduces a generic representation extraction module that uses semantics and domain knowledge to filter and encode static (HD maps) and dynamic (agent trajectories) scene elements into a unified, interpretable format.
• It formulates the environment representation as 2D/3D spatial-temporal semantic graphs (SG), where nodes represent semantic goals (e.g., lane changes, turns) and edges encode relational dependencies between agents and scene elements.
• A semantic graph network (SGN) is designed to reason over internal structural relations in the semantic graphs, leveraging the inductive biases of graph networks for permutation invariance and relational reasoning.
• The framework uses graph neural networks to process the semantic graphs, enabling end-to-end learning of interaction-aware behavior predictions from structured semantic inputs.
• The model is trained end-to-end on diverse scenarios and evaluated for zero-shot transferability from known domains (e.g., roundabouts) to unseen ones (e.g., T-intersections).
• Domain invariance is validated via t-SNE visualization of node features, showing high overlap across different road types and traffic conditions.

Experimental results

Research questions

• RQ1 Can a generic, semantics-based representation of driving environments be constructed to be invariant across diverse traffic scenarios and road structures?
• RQ2 Can semantic graph reasoning effectively model complex, interactive agent behaviors in dynamic traffic environments?
• RQ3 Does the proposed framework achieve state-of-the-art prediction performance while enabling zero-shot transferability to unseen driving scenarios?
• RQ4 To what extent do domain-invariant representations improve generalization across different traffic regulations and agent interaction patterns?

Key findings

• The proposed method achieves state-of-the-art performance in both prediction accuracy and confidence interval coverage across multiple benchmark scenarios.
• The model demonstrates strong zero-shot transferability, successfully generalizing from training on 8-way roundabouts to inference on unsignalized T-intersections without retraining.
• t-SNE visualization confirms that the generic semantic representations exhibit high domain invariance, with significant overlap in feature distributions across different road types.
• The semantic graph reasoning mechanism enables permutation invariance and robustness to varying numbers and orders of input agents and scene elements.
• The framework outperforms seven baseline methods in terms of prediction error and uncertainty calibration in complex, interactive urban scenarios.