[Paper Review] Latent Variable Sequential Set Transformers For Joint Multi-Agent Motion Prediction

Roger Girgis, Florian Golemo+6 more|arXiv (Cornell University)|Feb 19, 2021

Autonomous Vehicle Technology and Safety38 citations

TL;DR

AutoBots use latent-variable, transformer-based encoders/decoders to model sequences of sets for joint multi-agent motion prediction, yielding fast, multi-modal future trajectories with scene-consistent predictions. It achieves strong results on nuScenes and Argoverse while enabling single-GPU training.

ABSTRACT

Robust multi-agent trajectory prediction is essential for the safe control of robotic systems. A major challenge is to efficiently learn a representation that approximates the true joint distribution of contextual, social, and temporal information to enable planning. We propose Latent Variable Sequential Set Transformers which are encoder-decoder architectures that generate scene-consistent multi-agent trajectories. We refer to these architectures as "AutoBots". The encoder is a stack of interleaved temporal and social multi-head self-attention (MHSA) modules which alternately perform equivariant processing across the temporal and social dimensions. The decoder employs learnable seed parameters in combination with temporal and social MHSA modules allowing it to perform inference over the entire future scene in a single forward pass efficiently. AutoBots can produce either the trajectory of one ego-agent or a distribution over the future trajectories for all agents in the scene. For the single-agent prediction case, our model achieves top results on the global nuScenes vehicle motion prediction leaderboard, and produces strong results on the Argoverse vehicle prediction challenge. In the multi-agent setting, we evaluate on the synthetic partition of TrajNet++ dataset to showcase the model's socially-consistent predictions. We also demonstrate our model on general sequences of sets and provide illustrative experiments modelling the sequential structure of the multiple strokes that make up symbols in the Omniglot data. A distinguishing feature of AutoBots is that all models are trainable on a single desktop GPU (1080 Ti) in under 48h.

Motivation & Objective

• Model multi-agent motion prediction as sequences of sets with latent variables to capture multi-modality.
• Develop an encoder-decoder Transformer architecture with temporal and social attention.
• Enable single-pass decoding for multiple future modes via learnable seed parameters.
• Ensure permutation equivariance with respect to agents and sets.
• Demonstrate strong performance on nuScenes, Argoverse, TrajNet++, and Omniglot datasets.

Proposed method

• Encode a sequence of agent-sets using interleaved temporal and social multi-head self-attention blocks to produce a context tensor.
• Decode multiple future modes in parallel by using mode-specific learnable seed parameter matrices and repeated MABD/MAB layers conditioned on the encoder context.
• Incorporate additional environmental context through a CNN-derived vector M_i replicated across agents and timesteps.
• Train with a latent-variable objective that leverages a discrete Z and a variational-like Q to approximate posteriors, plus a mode entropy regularizer to encourage diverse outputs.
• Produce outputs as parameters of distributions (e.g., bivariate Gaussians) for each agent at each future timestep.
• Demonstrate permutation equivariance and compare inference speed to autoregressive baselines (single forward pass per mode).

Experimental results

Research questions

• RQ1 Can a latent-variable sequential set transformer (AutoBot) jointly model temporal and social interactions to generate consistent multi-agent futures?
• RQ2 Does single-pass decoding with learnable seed parameters capture multi-modal future modes efficiently without autoregressive sampling?
• RQ3 How does AutoBot perform on real-world autonomous driving benchmarks (nuScenes, Argoverse) and synthetic multi-agent datasets (TrajNet++)?
• RQ4 Is the model capable of producing diverse, realistic scene-consistent trajectories across tasks like Omniglot stroke sequences?
• RQ5 What is the computational efficiency of AutoBot compared to autoregressive or per-agent generation methods?

Key findings

• AutoBot-Ego achieves strong NuScenes results, best Min ADE (10) and low Off Road Rate, with competitive others.
• Ensembling three AutoBot-Ego models further improves performance on NuScenes.
• On Argoverse, AutoBot-Ego (valid) achieves Min ADE 0.73, Min FDE 1.10, Miss Rate 0.12; AutoBot-Ego (test) achieves 0.89 Min ADE (top-5) and 1.41 Min FDE (top-5).
• In TrajNet++ synthetic data, social attention in encoder/decoder reduces collisions and improves scene-level MinADE/MinFDE.
• Omniglot tasks show AutoBot generates more consistent/stylized strokes than LSTM baselines, including realistic completions under ambiguity.
• AutoBot-Ego can train on a single GPU (GTX 1080 Ti) in ~3 hours for nuScenes, and inference is faster than some autoregressive baselines by ~2x in cited comparisons.