[Paper Review] Exploiting map information for self-supervised learning in motion forecasting

Caio Azevedo, Thomas Gilles+2 more|arXiv (Cornell University)|Oct 10, 2022

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes a self-supervised learning auxiliary task, Map Trajectories, that leverages map graph connectivity to improve motion forecasting. By pretraining on diverse map-conditional trajectory generation, the method achieves up to 20.3% improvement in minFDE₆ and 33.3% in MissRate₆ across Argoverse, Interaction, and NuScenes, with state-of-the-art results in the Interaction challenge.

ABSTRACT

Inspired by recent developments regarding the application of self-supervised learning (SSL), we devise an auxiliary task for trajectory prediction that takes advantage of map-only information such as graph connectivity with the intent of improving map comprehension and generalization. We apply this auxiliary task through two frameworks - multitasking and pretraining. In either framework we observe significant improvement of our baseline in metrics such as $\mathrm{minFDE}_6$ (as much as 20.3%) and $\mathrm{MissRate}_6$ (as much as 33.3%), as well as a richer comprehension of map features demonstrated by different training configurations. The results obtained were consistent in all three data sets used for experiments: Argoverse, Interaction and NuScenes. We also submit our new pretrained model's results to the Interaction challenge and achieve $ extit{1st}$ place with respect to $\mathrm{minFDE}_6$ and $\mathrm{minADE}_6$.

Motivation & Objective

• To improve motion forecasting generalization by enhancing map comprehension through self-supervised learning.
• To investigate whether map-only information, such as lane connectivity, can serve as an effective auxiliary task in self-supervised learning for trajectory prediction.
• To evaluate the effectiveness of pretraining versus multitasking frameworks in leveraging map-based auxiliary tasks.
• To assess the robustness of the model to unseen map types, such as roundabouts, through pretraining on diverse map geometries.

Proposed method

• Designs a self-supervised auxiliary task, Map Trajectories, that generates diverse future trajectories from a given starting position on a map, using lanelet graph connectivity.
• Generates trajectories by varying speed, acceleration, and past/future trajectory length, with controlled noise to simulate real-world variability.
• Applies the auxiliary task in two frameworks: multitask learning (joint optimization with main task) and pretraining-fine-tuning (pretrain on map data, then fine-tune on trajectory data).
• Uses a base model with graph attention and graph convolutional networks to encode map and agent interactions, decoding full trajectory predictions and their probabilities.
• Employs a combined loss: smooth L1 loss on trajectory predictions and cross-entropy on prediction probabilities, with the main loss being a weighted sum of both.
• Pretraining is performed on large-scale map data without ground-truth trajectories, enabling the model to learn structural map representations.

Experimental results

Research questions

• RQ1 Can map-only information such as lane connectivity improve motion forecasting generalization through self-supervised learning?
• RQ2 Does pretraining on a map-based auxiliary task lead to better performance than multitask learning in motion forecasting?
• RQ3 How does the model generalize to unseen map types (e.g., roundabouts) when pretraining does not include such types?
• RQ4 What is the contribution of each trajectory generation step (e.g., variable speed, acceleration, noise) to the final performance of the self-supervised task?

Key findings

• The Map Trajectories pretraining framework achieved a 20.3% relative reduction in minFDE₆ and a 33.3% reduction in MissRate₆ on the Argoverse dataset.
• On the Interaction dataset, the method achieved 1st place in the challenge with the lowest minFDE₆ and minADE₆ scores, demonstrating strong generalization.
• Even when pretraining data excluded roundabouts, the model achieved a 29.3% relative improvement in minFDE₆ on roundabout maps, indicating robust map topology generalization.
• The ablation study showed that each trajectory generation step—especially fixed acceleration and noise—improved both direct pretraining performance and fine-tuned metrics.
• The raw pretraining performance strongly correlated with fine-tuned performance, suggesting that pretraining quality is a reliable proxy for downstream success.
• The pretraining framework outperformed multitasking, indicating that pretraining may be more effective than joint optimization for this specific auxiliary task.