[Paper Review] Perceive, Interact, Predict: Learning Dynamic and Static Clues for End-to-End Motion Prediction

Bo Jiang, Shaoyu Chen+8 more|arXiv (Cornell University)|Dec 5, 2022

Autonomous Vehicle Technology and Safety4 citations

TL;DR

This paper proposes PIP, an end-to-end Transformer-based framework that jointly performs online mapping, object detection, and motion prediction using only video input, eliminating the need for human-annotated HD maps or historical tracking. By leveraging map, agent, and mode queries with a differentiable multi-task interaction scheme, PIP achieves state-of-the-art performance with 0.258 EPA, outperforming both tracking-based and HD-map-based methods.

ABSTRACT

Motion prediction is highly relevant to the perception of dynamic objects and static map elements in the scenarios of autonomous driving. In this work, we propose PIP, the first end-to-end Transformer-based framework which jointly and interactively performs online mapping, object detection and motion prediction. PIP leverages map queries, agent queries and mode queries to encode the instance-wise information of map elements, agents and motion intentions, respectively. Based on the unified query representation, a differentiable multi-task interaction scheme is proposed to exploit the correlation between perception and prediction. Even without human-annotated HD map or agent's historical tracking trajectory as guidance information, PIP realizes end-to-end multi-agent motion prediction and achieves better performance than tracking-based and HD-map-based methods. PIP provides comprehensive high-level information of the driving scene (vectorized static map and dynamic objects with motion information), and contributes to the downstream planning and control. Code and models will be released for facilitating further research.

Motivation & Objective

• To address the limitations of HD-map-dependent and tracking-based motion prediction methods that rely on costly, static, or outdated map data.
• To enable end-to-end learning of motion prediction directly from video input, without requiring human-annotated HD maps or historical agent trajectories.
• To jointly optimize perception and prediction through a unified query-based framework that models interactions between dynamic agents, static map elements, and motion intentions.
• To develop a fully differentiable, multi-task interaction mechanism that exploits correlations between perception and prediction for improved accuracy.

Proposed method

• PIp employs three types of queries—map queries, agent queries, and mode queries—to encode instance-wise representations of static map elements, dynamic agents, and motion intentions, respectively.
• A Motion Interactor module enables explicit and implicit interaction between agent-centric representations and map features, using attention mechanisms to fuse map context and agent state.
• The framework performs agent-wise normalization by switching from ego-centric to agent-centric coordinate systems to reduce variance in predictions.
• Map features are filtered based on confidence scores and spatial proximity to each agent, ensuring relevant map context is used in prediction.
• The model is trained end-to-end with a multi-task loss combining detection, mapping, and motion prediction objectives, with learnable loss weights.
• VectorNet is used to further encode map features into instance-level representations for enhanced motion-map interaction.

Experimental results

Research questions

• RQ1 Can an end-to-end framework jointly learn online mapping, object detection, and motion prediction from raw video without relying on pre-constructed HD maps?
• RQ2 How can perception and prediction be effectively correlated through a differentiable, multi-task interaction mechanism in a unified query-based Transformer architecture?
• RQ3 To what extent can a video-only input achieve motion prediction performance comparable to or better than methods using HD maps or historical tracking data?
• RQ4 What is the impact of explicit and implicit map-agent interaction on trajectory prediction accuracy and diversity?

Key findings

• PIP achieves a 0.258 EPA, surpassing the previous state-of-the-art method [11], demonstrating superior motion prediction performance without HD maps or tracking.
• PIP reduces minADE by 0.91m and minFDE by 1.17m compared to the previous SOTA, indicating significant improvement in trajectory accuracy.
• The model's performance using predicted vectorized maps is nearly on par with that using ground-truth HD maps, with only a 0.01m gap in minADE, showing robustness to map prediction errors.
• Ablation studies confirm the effectiveness of the multi-task interaction design, particularly the agent-wise normalization and confidence-based map filtering.
• Qualitative results show that PIP learns meaningful agent-map correlations and generates diverse, plausible multi-modal predictions, especially for merging vehicles.
• The framework outputs comprehensive high-level scene representations—vectorized maps and motion-aware object detections—enabling downstream planning and control.