[Paper Review] Multimodal Motion Prediction with Stacked Transformers

Yicheng Liu, Jinghuai Zhang+3 more|arXiv (Cornell University)|Mar 22, 2021

Autonomous Vehicle Technology and Safety36 references21 citations

TL;DR

This paper proposes mmTransformer, a stacked transformer-based model for multimodal motion prediction that uses fixed trajectory proposals and a region-based training strategy to enhance diversity and accuracy. By refining proposals through hierarchical attention and training each proposal to specialize in a spatial region, the model achieves state-of-the-art performance on the Argoverse benchmark, outperforming prior methods in both trajectory accuracy and mode coverage.

ABSTRACT

Predicting multiple plausible future trajectories of the nearby vehicles is crucial for the safety of autonomous driving. Recent motion prediction approaches attempt to achieve such multimodal motion prediction by implicitly regularizing the feature or explicitly generating multiple candidate proposals. However, it remains challenging since the latent features may concentrate on the most frequent mode of the data while the proposal-based methods depend largely on the prior knowledge to generate and select the proposals. In this work, we propose a novel transformer framework for multimodal motion prediction, termed as mmTransformer. A novel network architecture based on stacked transformers is designed to model the multimodality at feature level with a set of fixed independent proposals. A region-based training strategy is then developed to induce the multimodality of the generated proposals. Experiments on Argoverse dataset show that the proposed model achieves the state-of-the-art performance on motion prediction, substantially improving the diversity and the accuracy of the predicted trajectories. Demo video and code are available at https://decisionforce.github.io/mmTransformer.

Motivation & Objective

• To address the challenge of learning multiple plausible future trajectories in autonomous driving under data scarcity and uncertainty.
• To overcome limitations in probabilistic and proposal-based methods, such as mode collapse and dependency on heuristic anchors.
• To design a deep learning framework that explicitly models multimodality at the feature level without relying on predefined priors.
• To ensure each prediction proposal specializes in a distinct mode by enforcing region-specific learning during training.

Proposed method

• The model employs a stacked transformer architecture with multiple encoder-decoder modules to hierarchically aggregate contextual features from past trajectories, road layouts, and social interactions.
• Trajectory proposals are randomly initialized and refined via multi-head self-attention and cross-attention mechanisms to generate diverse predictions.
• A region-based training strategy (RTS) partitions the surrounding space into M regions, assigning each proposal to one region and only optimizing proposals in the ground truth region during training.
• Each proposal is associated with a confidence score via a feed-forward network, and a classification loss is applied to encourage high scores for proposals in the correct region.
• The model uses spatial positional encodings and Xavier initialization, with data augmentation via horizontal flipping and masking of early trajectory steps.
• During inference, non-maximum suppression (NMS) filters overlapping trajectories based on confidence scores and IoU thresholds.

Experimental results

Research questions

• RQ1 Can a transformer-based architecture effectively model multimodal motion prediction by learning from fixed, independent trajectory proposals?
• RQ2 How can a training strategy be designed to ensure each proposal learns a distinct mode despite only one ground truth per scene?
• RQ3 Does region-based training improve the diversity and accuracy of predicted trajectories compared to standard training paradigms?
• RQ4 Can stacked transformers effectively capture long-range dependencies and contextual interactions in complex driving scenes?
• RQ5 To what extent does the proposed method outperform existing probabilistic and proposal-based approaches on real-world benchmarks?

Key findings

• The proposed mmTransformer achieved the 1st rank on the Argoverse motion prediction leaderboard as of November 16, 2020, demonstrating state-of-the-art performance.
• The region-based training strategy significantly improved mode coverage, with 85.3% of predictions falling within the correct region when M=N=6.
• The model achieved a minimum missing rate of 12.1% in the diagonal of the MR matrix, indicating strong specialization of proposals to assigned regions.
• The ablation study showed that the region-based training strategy improved the average displacement error by 12.4% compared to standard training.
• The model maintained high performance across various scenarios, including complex intersections and high-density traffic, due to its robust feature aggregation and proposal specialization.