[Paper Review] Multi-modal Motion Prediction with Transformer-based Neural Network for Autonomous Driving

Zhiyu Huang, Xiaoyu Mo+1 more|arXiv (Cornell University)|Sep 14, 2021

Autonomous Vehicle Technology and Safety4 citations

TL;DR

This paper proposes a Transformer-based motion prediction model that uses multi-modal attention to capture diverse agent-map interactions for multi-modal trajectory forecasting. By modifying multi-head attention into independent attention modes per trajectory, the model achieves state-of-the-art accuracy on the Argoverse benchmark with a compact, interpretable architecture.

ABSTRACT

Predicting the behaviors of other agents on the road is critical for autonomous driving to ensure safety and efficiency. However, the challenging part is how to represent the social interactions between agents and output different possible trajectories with interpretability. In this paper, we introduce a neural prediction framework based on the Transformer structure to model the relationship among the interacting agents and extract the attention of the target agent on the map waypoints. Specifically, we organize the interacting agents into a graph and utilize the multi-head attention Transformer encoder to extract the relations between them. To address the multi-modality of motion prediction, we propose a multi-modal attention Transformer encoder, which modifies the multi-head attention mechanism to multi-modal attention, and each predicted trajectory is conditioned on an independent attention mode. The proposed model is validated on the Argoverse motion forecasting dataset and shows state-of-the-art prediction accuracy while maintaining a small model size and a simple training process. We also demonstrate that the multi-modal attention module can automatically identify different modes of the target agent's attention on the map, which improves the interpretability of the model.

Motivation & Objective

• Address the challenge of modeling complex, multi-modal interactions between autonomous vehicles and surrounding agents in dynamic traffic environments.
• Improve trajectory prediction accuracy by explicitly modeling the target agent's attention to different map waypoints, especially in scenarios with multiple behavioral intentions.
• Enhance model interpretability by enabling the attention mechanism to automatically identify and condition on distinct map segments (e.g., turning lanes, straight paths).
• Develop a lightweight, efficient architecture with minimal training complexity while maintaining high performance on real-world driving data.
• Investigate the impact of map representation (lane vs. waypoint) and attention mechanism design on prediction performance and interpretability.

Proposed method

• Represent the driving scene using a waypoint-based vector map, where each lane is encoded as a sequence of spatial waypoints for high-resolution map representation.
• Model interactions among agents using a graph-based structure, treating each agent as a node connected to the target agent, and apply a multi-head self-attention Transformer encoder to extract relational features.
• Introduce a novel multi-modal attention mechanism that replaces standard multi-head attention by treating each attention head as an independent mode, each conditioning a separate predicted trajectory.
• Train the model using a loss function that only updates the attention head producing the closest trajectory to the ground truth, preserving diversity among predicted modes.
• Use two cross-attention Transformer layers: one for agent-agent interaction modeling and another for agent-map attention, with shared positional encoding for temporal and spatial consistency.
• Apply a simple trajectory decoder head to generate future trajectories from the final hidden states of the multi-modal attention layer.

Experimental results

Research questions

• RQ1 Can a modified multi-modal attention mechanism in a Transformer-based architecture improve multi-modal motion prediction accuracy compared to standard multi-head attention?
• RQ2 How does the use of waypoint-level map representation compare to lane-level representation in terms of prediction accuracy and information retention?
• RQ3 To what extent can the attention scores of individual attention heads be interpreted as the model’s focus on specific map segments (e.g., turning lanes) for different predicted behaviors?
• RQ4 Does the proposed method achieve competitive performance with a simpler training process and smaller model size compared to existing state-of-the-art models?
• RQ5 How does the proposed multi-modal attention mechanism contribute to the diversity and correctness of predicted trajectories in complex urban driving scenarios?

Key findings

• The proposed model achieves the lowest minADE (0.8372 m) and minFDE (1.2905 m) on the Argoverse test set among all compared methods, indicating superior trajectory prediction accuracy.
• The model demonstrates improved interpretability: visualization shows that attention heads naturally focus on relevant map waypoints—e.g., left-turn lanes during left-turn predictions—confirming the model learns meaningful behavioral intentions.
• Using waypoint-level map representation yields better performance than lane-level representation, reducing information loss and improving resolution of map features.
• The multi-modal attention mechanism significantly outperforms an ensemble-based multi-modal prediction method (minADE: 0.8372 vs. 0.8512), indicating its effectiveness in modeling diverse behaviors.
• Despite achieving state-of-the-art accuracy, the model maintains a small size and simple training process, with only two cross-attention Transformer layers, enabling fast inference and easier deployment.
• The model’s brier-minFDE (1.9393) and miss rate (0.1429) are slightly worse than HOME [24], but this is attributed to the loss function not being optimized for miss rate, not a fundamental limitation of the architecture.