[Paper Review] Motion Transformer with Global Intention Localization and Local Movement Refinement

Shaoshuai Shi, Li Jiang+2 more|arXiv (Cornell University)|Sep 27, 2022

Autonomous Vehicle Technology and Safety74 citations

TL;DR

The paper introduces Motion TRansformer (MTR), a transformer-based multimodal motion prediction framework that uses a small set of learnable motion query pairs to localize global intentions and refine local movements, achieving state-of-the-art results on Waymo Open Motion Dataset without dense goal candidates.

ABSTRACT

Predicting multimodal future behavior of traffic participants is essential for robotic vehicles to make safe decisions. Existing works explore to directly predict future trajectories based on latent features or utilize dense goal candidates to identify agent's destinations, where the former strategy converges slowly since all motion modes are derived from the same feature while the latter strategy has efficiency issue since its performance highly relies on the density of goal candidates. In this paper, we propose Motion TRansformer (MTR) framework that models motion prediction as the joint optimization of global intention localization and local movement refinement. Instead of using goal candidates, MTR incorporates spatial intention priors by adopting a small set of learnable motion query pairs. Each motion query pair takes charge of trajectory prediction and refinement for a specific motion mode, which stabilizes the training process and facilitates better multimodal predictions. Experiments show that MTR achieves state-of-the-art performance on both the marginal and joint motion prediction challenges, ranking 1st on the leaderboards of Waymo Open Motion Dataset. The source code is available at https://github.com/sshaoshuai/MTR.

Motivation & Objective

• Motivate robust multimodal motion prediction for autonomous driving by avoiding dense goal candidates and slow convergence.
• Propose a unified framework that combines global intention localization with local movement refinement via motion query pairs.
• Incorporate an auxiliary dense future prediction task to enrich future interaction context.
• Demonstrate state-of-the-art performance on marginal and joint motion prediction benchmarks of WOMD.

Proposed method

• Introduce motion query pairs consisting of static intention queries for global localization and dynamic searching queries for local refinement.
• Use a transformer encoder with local self-attention to encode scene context from agent histories and road maps.
• Employ a dense future prediction auxiliary task to predict future trajectories and velocities for all agents as additional context.
• Implement a dynamic map collection module to gather trajectory-aligned map features for each motion mode.
• Apply a Gaussian Mixture Model head to represent multimodal future distributions and use hard assignment for training.
• Train end-to-end with auxiliary L1 loss and negative log-likelihood loss across decoder layers.

Experimental results

Research questions

• RQ1 Can a small set of learnable motion query pairs effectively cover multimodal future motions without dense goal candidates?
• RQ2 Does global intention localization plus local movement refinement improve stability and accuracy of predictions?
• RQ3 Does incorporating a dense future prediction task improve scene-consistent trajectory generation?
• RQ4 How does local self-attention in the encoder affect context modeling and memory efficiency in large map contexts?

Key findings

• MTR achieves state-of-the-art performance on marginal motion prediction in WOMD, with notable gains in mAP over ensemble-free baselines.
• MTR also outperforms prior methods on joint motion prediction benchmarks, achieving higher mAP and lower miss rates.
• Static intention queries with intention points provide substantial mAP improvements compared to latent learnable embeddings, highlighting mode-specific querying benefits.
• Dense future prediction modules significantly improve trajectory quality by capturing interactions among future trajectories.
• Local self-attention enables better performance and memory efficiency than global attention when using large map contexts.
• Ensemble variants and end-to-end MTR-e2e show strong performance, with MTR-Advanced-ens achieving top rankings on test sets.