[Paper Review] Heterogeneous Edge-Enhanced Graph Attention Network For Multi-Agent Trajectory Prediction

Xiaoyu Mo, Yang Xing+1 more|arXiv (Cornell University)|Jun 14, 2021

Autonomous Vehicle Technology and SafetyEngineering35 references17 citations

TL;DR

This paper proposes a three-channel framework with a novel Heterogeneous Edge-enhanced Graph Attention Network (HEAT) for simultaneous multi-agent trajectory prediction in complex urban and highway scenarios. By modeling heterogeneous agent dynamics, edge-featured directed graphs for inter-agent interactions, and a gated map selector for shared map representation, HEAT achieves state-of-the-art performance with 0.68m RMSE at 1-second horizon and 2.05m at 5-second horizon on the INTERACTION dataset.

ABSTRACT

Simultaneous trajectory prediction for multiple heterogeneous traffic participants is essential for the safe and efficient operation of connected automated vehicles under complex driving situations in the real world. The multi-agent prediction task is challenging, as the motions of traffic participants are affected by many factors, including their individual dynamics, their interactions with surrounding agents, the traffic infrastructures, and the number and modalities of the target agents. To further advance the trajectory prediction techniques, in this work we propose a three-channel framework together with a novel Heterogeneous Edge-enhanced graph ATtention network (HEAT), which is able to deal with the heterogeneity of the target agents and traffic participants involved. Specifically, the agent's dynamics are extracted from their historical states using type-specific encoders. The inter-agent interactions are represented with a directed edge-featured heterogeneous graph, and then interaction features are extracted using the proposed HEAT network. Besides, the map features are shared across all agents by introducing a selective gate mechanism. And finally, the trajectories of multi-agent are executed simultaneously. Validations using both urban and highway driving datasets show that the proposed model can realize simultaneous trajectory predictions for multiple agents under complex traffic situations, and achieve state-of-the-art performance with respect to prediction accuracy, demonstrating its feasibility and effectiveness.

Motivation & Objective

• Address the challenge of simultaneous trajectory prediction for multiple heterogeneous traffic agents (vehicles, pedestrians, cyclists) in complex driving environments.
• Overcome limitations of existing methods that rely on shared coordinate systems, which are sensitive to translation and rotation and restrict modeling flexibility.
• Improve interaction modeling by representing inter-agent interactions through a directed, edge-featured heterogeneous graph that captures agent type and interaction dynamics.
• Enable selective, shared map representation across agents using a gate mechanism to improve efficiency and generalization without storing per-agent maps.
• Achieve high prediction accuracy across diverse scenarios, including both urban and highway environments, while handling arbitrary numbers of agents and varying modalities.

Proposed method

• Employ a three-channel framework: one for agent-specific dynamics (using type-specific encoders in individual coordinate systems), one for interaction modeling (via a heterogeneous edge-featured graph), and one for map representation (with a gated selector).
• Represent inter-agent interactions as a directed, heterogeneous graph where nodes denote agents of different types (e.g., vehicle, pedestrian) and edges carry attributes such as relative position, velocity, and interaction type.
• Design the Heterogeneous Edge-enhanced Graph Attention Network (HEAT) to process the graph by attending to both node and edge features, enabling type-aware message passing across heterogeneous agent types.
• Introduce a selective gate mechanism that dynamically controls how much map information each agent receives, allowing efficient and adaptive map feature sharing across all agents.
• Use a shared top-view map encoded via a CNN, with the gate mechanism modulating the map input based on agent-specific relevance, reducing redundancy and improving generalization.
• Train the model end-to-end using a multi-task loss combining trajectory prediction and interaction modeling objectives, enabling joint optimization of dynamics, interactions, and map features.

Experimental results

Research questions

• RQ1 Can a graph-based method effectively model heterogeneous interactions among diverse traffic participants (vehicles, pedestrians, cyclists) in multi-agent trajectory prediction?
• RQ2 How does using individual coordinate systems per agent improve prediction robustness compared to shared coordinate systems in dynamic traffic environments?
• RQ3 To what extent does incorporating edge features (e.g., relative motion, interaction type) in a heterogeneous graph improve interaction modeling over standard GATs?
• RQ4 Can a gate-based mechanism effectively share a single map representation across multiple agents without degrading performance or increasing computational cost?
• RQ5 Does the proposed framework achieve state-of-the-art performance in both short- and long-term trajectory prediction across diverse real-world driving scenarios?

Key findings

• The proposed HEAT model achieves a 1-second prediction RMSE of 0.68 meters on the INTERACTION dataset, outperforming prior methods such as CS-LSTM (0.61m) and CNN-LSTM (0.64m) in the same setting.
• At a 5-second prediction horizon, HEAT achieves 2.05m RMSE, significantly outperforming baseline GAT-based models (e.g., 4.87m for GAT-R-1) and matching or exceeding state-of-the-art methods like MATF-GAN and Scale-Net.
• HEAT-R (with shared map and HEAT interaction module) achieves 0.68m RMSE at 1s and 2.05m at 5s, demonstrating robustness and scalability across diverse agent counts and types.
• Ablation studies show that HEAT-1 (with HEAT) achieves 0.6590m RMSE at 1s, significantly outperforming GAT-1 (0.7808m) and GAT-R-1 (0.8228m), proving the superiority of the proposed interaction modeling approach.
• The gated map selector enables effective map sharing, improving generalization and reducing computational overhead compared to per-agent map storage or naive global sharing.
• Visualizations confirm that HEAT accurately predicts trajectories for multiple heterogeneous agents in complex scenarios, including merging, crossing, and high-density traffic.