[Paper Review] Multi-agent Trajectory Prediction with Fuzzy Query Attention

Nitin Kamra, Hao Zhu+3 more|arXiv (Cornell University)|Oct 29, 2020

Autonomous Vehicle Technology and Safety32 references19 citations

TL;DR

This paper proposes Fuzzy Query Attention (FQA), a novel attention mechanism that models multi-agent trajectory prediction by learning continuous-valued (fuzzy) decisions for agent interactions, capturing motion inductive biases like inertia, relative motion, and intent. FQA achieves state-of-the-art performance across diverse domains—including human crowds, freeway traffic, physics simulations, and NBA player trajectories—by jointly learning interaction decisions and their dynamic responses via a relational graph architecture.

ABSTRACT

Trajectory prediction for scenes with multiple agents and entities is a challenging problem in numerous domains such as traffic prediction, pedestrian tracking and path planning. We present a general architecture to address this challenge which models the crucial inductive biases of motion, namely, inertia, relative motion, intents and interactions. Specifically, we propose a relational model to flexibly model interactions between agents in diverse environments. Since it is well-known that human decision making is fuzzy by nature, at the core of our model lies a novel attention mechanism which models interactions by making continuous-valued (fuzzy) decisions and learning the corresponding responses. Our architecture demonstrates significant performance gains over existing state-of-the-art predictive models in diverse domains such as human crowd trajectories, US freeway traffic, NBA sports data and physics datasets. We also present ablations and augmentations to understand the decision-making process and the source of gains in our model.

Motivation & Objective

• To address the challenge of modeling complex, continuous-valued interactions in multi-agent trajectory prediction, which are inherently fuzzy rather than binary.
• To develop a generalizable architecture that captures key inductive biases of motion: inertia, relative motion, intent, and interactions.
• To improve predictive performance across diverse domains by modeling interactions through a novel attention mechanism that learns fuzzy decisions and their dynamic responses.
• To enable integration of human knowledge into the model via manually inserted fuzzy decisions to enhance performance and interpretability.

Proposed method

• FQA employs a graph-based architecture where agents are nodes and interactions are modeled via learned attention mechanisms.
• The core mechanism uses query and key vectors to compute continuous-valued (fuzzy) attention scores, representing the degree of interaction between agent pairs.
• These fuzzy decisions are used to compute weighted message passing, learning how each agent is affected by influencing agents.
• The model uses a relational message-passing framework that dynamically updates agent representations based on fuzzy interaction strengths.
• It supports human-in-the-loop augmentation by allowing insertion of known fuzzy decisions (e.g., 'agent A is slowing down') to guide learning.
• The architecture is trained end-to-end using trajectory data, with loss functions optimized for prediction accuracy across multiple datasets.

Experimental results

Research questions

• RQ1 Can continuous-valued (fuzzy) decision-making improve multi-agent trajectory prediction compared to binary or discrete interaction modeling?
• RQ2 How effectively can a fuzzy attention mechanism capture complex, real-world interactions in diverse domains such as traffic, crowds, and physics?
• RQ3 To what extent do fuzzy decisions alone, without other features, predict interaction patterns in trajectory data?
• RQ4 Can human-provided fuzzy decisions be effectively integrated into the model to improve performance?
• RQ5 How does FQA compare to state-of-the-art models across varied datasets with different interaction dynamics?

Key findings

• FQA achieves significant performance gains over state-of-the-art models on human crowd trajectories, US freeway traffic, Newtonian physics simulations, electrostatic particle motion, and NBA player tracking.
• Fuzzy decisions alone—without other input features—were highly predictive of interactions, demonstrating their intrinsic representational power.
• On the NGSIM freeway dataset, incorporating human-knowledge as fuzzy decisions reduced RMSE, indicating improved modeling of boundary and sub-scene interaction effects.
• FQA accurately models rare but critical events like inter-agent collisions and boundary interactions, while baselines often miss or misrepresent them.
• FQA produced sharper, more realistic trajectories in chaotic systems like charged particles under Coulombic forces, outperforming baselines in capturing high-acceleration dynamics.
• Despite strong performance, FQA still faces challenges on the NBA dataset due to high intent-dependency and long-term unpredictability, suggesting the need for complementary intent modeling.