[Paper Review] PedFormer: Pedestrian Behavior Prediction via Cross-Modal Attention Modulation and Gated Multitask Learning

Amir Rasouli, Iuliia Kotseruba|arXiv (Cornell University)|Oct 14, 2022

Autonomous Vehicle Technology and Safety4 citations

TL;DR

PedFormer proposes a cross-modal Transformer-based framework with gated multitask learning to jointly predict pedestrian trajectories and crossing actions using ego-centric video, motion, and scene data. By integrating cross-modal attention, dynamic interaction modeling, and a gated hybrid decoder, it achieves state-of-the-art performance, improving SOTA by up to 22% on trajectory and 13% on action prediction metrics.

ABSTRACT

Predicting pedestrian behavior is a crucial task for intelligent driving systems. Accurate predictions require a deep understanding of various contextual elements that potentially impact the way pedestrians behave. To address this challenge, we propose a novel framework that relies on different data modalities to predict future trajectories and crossing actions of pedestrians from an ego-centric perspective. Specifically, our model utilizes a cross-modal Transformer architecture to capture dependencies between different data types. The output of the Transformer is augmented with representations of interactions between pedestrians and other traffic agents conditioned on the pedestrian and ego-vehicle dynamics that are generated via a semantic attentive interaction module. Lastly, the context encodings are fed into a multi-stream decoder framework using a gated-shared network. We evaluate our algorithm on public pedestrian behavior benchmarks, PIE and JAAD, and show that our model improves state-of-the-art in trajectory and action prediction by up to 22% and 13% respectively on various metrics. The advantages brought by components of our model are investigated via extensive ablation studies.

Motivation & Objective

• To address the challenge of accurately predicting pedestrian behavior in urban driving scenarios by jointly modeling future trajectories and crossing actions.
• To improve prediction performance by leveraging multimodal data—visual, motion, and scene context—through cross-modal attention mechanisms.
• To enhance interaction modeling between pedestrians and traffic agents by incorporating dynamic ego-vehicle and pedestrian motion into semantic scene representations.
• To design a robust decoding architecture that effectively shares knowledge across trajectory and action prediction tasks while minimizing interference.
• To validate the effectiveness of each component through extensive ablation studies on public benchmarks.

Proposed method

• The model uses a cross-modal Transformer encoder to learn dependencies between visual, motion, and scene modalities using query, key, and value representations from different data types.
• An interaction modeling module employs a dual-attention mechanism to generate context-aware representations by fusing pedestrian and ego-vehicle dynamics with semantic scene features.
• The interaction representations are concatenated with modality-encoding outputs and fed into a gated hybrid multi-stream decoder to predict both trajectories and actions.
• The gated decoder uses a learnable gating mechanism to control information flow from a shared representation to task-specific decoders, enabling selective feature utilization.
• The framework employs a multitask learning objective where trajectory and action predictions are optimized jointly with shared and task-specific heads.
• The model is trained end-to-end on the PIE and JAAD datasets using standard metrics including ADE, FDE, AUC, F1, and IoU.

Experimental results

Research questions

• RQ1 How does cross-modal attention modulation between visual, motion, and scene data improve pedestrian behavior prediction?
• RQ2 To what extent does incorporating dynamic interaction modeling between pedestrians and ego-vehicles enhance prediction accuracy?
• RQ3 How does a gated hybrid decoder architecture compare to shared or task-specific decoders in multitask pedestrian prediction?
• RQ4 What is the contribution of each component—cross-modal attention, interaction modeling, and gated decoding—to overall performance?
• RQ5 Can joint prediction of trajectories and actions lead to better performance than single-task approaches?

Key findings

• PedFormer achieves state-of-the-art performance on both PIE and JAAD benchmarks, improving SOTA by up to 22% on trajectory prediction metrics (ADE, FDE, ARB, FRB).
• The inclusion of cross-modal attention leads to a significant performance boost, with improvements of up to 12% on trajectory metrics and 3% on action metrics (AUC, F1) compared to baselines without it.
• The proposed interaction modeling module, which fuses motion and semantic scene features, improves trajectory prediction by up to 12% and action prediction by 3% over ablation baselines.
• The gated hybrid decoder outperforms task-based, shared, and bifold decoders, achieving up to 12% improvement on trajectory metrics and 4% on action metrics.
• Ablation studies confirm that removing motion encoding or using a simple 1×1 convolution instead of attention-based interaction modeling leads to performance degradation, highlighting the importance of dynamic context.
• Qualitative results show that PedFormer produces more accurate and robust predictions, especially during high-ego-motion events like turns or accelerations.