[Paper Review] Action-based Contrastive Learning for Trajectory Prediction

Marah Halawa, Olaf Hellwich+1 more|arXiv (Cornell University)|Jul 18, 2022

Autonomous Vehicle Technology and Safety4 citations

TL;DR

This paper proposes an action-based contrastive learning framework that improves trajectory prediction by leveraging pedestrian action labels during training to guide trajectory embedding space learning. By introducing a novel action-based contrastive loss and using synthetic trajectories from a CVAE for enhanced sampling, the method achieves state-of-the-art performance on three first-person trajectory prediction benchmarks, outperforming existing approaches by learning more behavior-aware representations.

ABSTRACT

Trajectory prediction is an essential task for successful human robot interaction, such as in autonomous driving. In this work, we address the problem of predicting future pedestrian trajectories in a first person view setting with a moving camera. To that end, we propose a novel action-based contrastive learning loss, that utilizes pedestrian action information to improve the learned trajectory embeddings. The fundamental idea behind this new loss is that trajectories of pedestrians performing the same action should be closer to each other in the feature space than the trajectories of pedestrians with significantly different actions. In other words, we argue that behavioral information about pedestrian action influences their future trajectory. Furthermore, we introduce a novel sampling strategy for trajectories that is able to effectively increase negative and positive contrastive samples. Additional synthetic trajectory samples are generated using a trained Conditional Variational Autoencoder (CVAE), which is at the core of several models developed for trajectory prediction. Results show that our proposed contrastive framework employs contextual information about pedestrian behavior, i.e. action, effectively, and it learns a better trajectory representation. Thus, integrating the proposed contrastive framework within a trajectory prediction model improves its results and outperforms state-of-the-art methods on three trajectory prediction benchmarks [31, 32, 26].

Motivation & Objective

• To improve trajectory prediction in first-person view settings by incorporating pedestrian action information into the learning process.
• To address the challenge of limited negative and positive contrastive samples in trajectory data, especially when conditioning on action classes.
• To develop a sampling strategy that avoids heuristic-based negative mining by leveraging learned latent distributions from a CVAE.
• To demonstrate that action-aware trajectory representations lead to better generalization and prediction performance.
• To enable semi-supervised training by reducing dependency on action labels for synthetic data generation.

Proposed method

• Proposes an action-based contrastive loss that encourages trajectories of pedestrians performing the same action to have closer embeddings in the feature space.
• Uses a Conditional Variational Autoencoder (CVAE) to generate synthetic trajectory samples, which are used to increase both positive and negative contrastive pairs.
• Integrates the action-based contrastive loss as a regularizer to the main trajectory prediction loss, with a learnable weighting hyperparameter β.
• Employs a novel sampling strategy that treats trajectories from the same action class as positive samples and different action classes as negative samples.
• Leverages the CVAE's learned latent space to sample diverse and realistic trajectories, improving the diversity and quality of contrastive pairs.
• Trains the model end-to-end with both real and synthetic trajectories, enabling effective contrastive learning without requiring action labels for synthetic data.

Experimental results

Research questions

• RQ1 Can incorporating pedestrian action information into the contrastive learning process improve trajectory prediction performance?
• RQ2 Does using CVAE-generated synthetic trajectories enhance the quality and diversity of positive and negative contrastive samples?
• RQ3 How does the proposed action-based contrastive loss compare to standard contrastive losses like SimCLR in trajectory prediction tasks?
• RQ4 What is the optimal balance between the main prediction loss and the contrastive regularization loss, as controlled by the hyperparameter β?
• RQ5 Can the framework reduce reliance on action labels during training by using synthetic data from a CVAE?

Key findings

• The proposed action-based contrastive learning framework (ABC+) achieves state-of-the-art performance on three first-person trajectory prediction benchmarks: PIE, JAAD, and TITAN.
• On the TITAN dataset, ABC+ reduces C-FDE from 843 to 165 and C-ADE(1.5) from 434 to 302, demonstrating significant performance gains.
• The ablation study shows that ABC+ outperforms ABC (without synthetic samples), confirming the effectiveness of the CVAE-based sampling strategy.
• Setting the hyperparameter β to 0.75 yields the best results across all datasets, indicating optimal trade-off between prediction and contrastive loss.
• The action-based contrastive loss outperforms the standard SimCLR contrastive loss, proving that action-aware supervision improves trajectory representation learning.
• The framework enables semi-supervised training, as synthetic samples from the CVAE do not require action labels, reducing dependency on annotated action data.