[Paper Review] Multiple Futures Prediction

Yichuan Tang, Ruslan Salakhutdinov|arXiv (Cornell University)|Nov 4, 2019

Autonomous Vehicle Technology and SafetyEngineering119 citations

TL;DR

MFP is a probabilistic, end-to-end framework with discrete latent variables that jointly predicts multimodal, interactive future trajectories for arbitrarily many agents, with closed-form probabilities and hypothetical rollouts demonstrated on driving datasets.

ABSTRACT

Temporal prediction is critical for making intelligent and robust decisions in complex dynamic environments. Motion prediction needs to model the inherently uncertain future which often contains multiple potential outcomes, due to multi-agent interactions and the latent goals of others. Towards these goals, we introduce a probabilistic framework that efficiently learns latent variables to jointly model the multi-step future motions of agents in a scene. Our framework is data-driven and learns semantically meaningful latent variables to represent the multimodal future, without requiring explicit labels. Using a dynamic attention-based state encoder, we learn to encode the past as well as the future interactions among agents, efficiently scaling to any number of agents. Finally, our model can be used for planning via computing a conditional probability density over the trajectories of other agents given a hypothetical rollout of the 'self' agent. We demonstrate our algorithms by predicting vehicle trajectories of both simulated and real data, demonstrating the state-of-the-art results on several vehicle trajectory datasets.

Motivation & Objective

• Motivate robust, safe decision making in dynamic, multi-agent environments by forecasting multiple plausible futures.
• Develop a scalable model that captures inter-agent interactions and context without fixed agent counts or explicit mode labels.
• Learn semantically meaningful latent modes directly from data to represent diverse future behaviours.
• Enable planning via hypothetical inference by evaluating conditional trajectory distributions given imagined agent rollouts.

Proposed method

• Propose a sequential probabilistic latent-variable model (MFP) that factors the joint future across time and agents to enable scalability to arbitrary N.
• Use discrete latent variables z^n (one per agent) drawn from Multinoulli(K) to capture multimodal future modes without labeled intents.
• Implement per-agent PoV encoders, dynamic attention-based state encoding, and per-agent decoding RNNs sharing weights to predict future trajectories.
• Provide a tractable learning objective via an EM-like optimization using the ELBO, with exact posterior computation for the discrete Z due to small K.
• Allow joint, interactive rollouts by conditioning each agent’s future on the predicted futures of others, enabling hypothetical inference (conditioning on ego trajectories).
• Output per-agent future as a 5D Gaussian (mu_x, mu_y, sigma_x, sigma_y, rho) for tractable likelihoods and planning.

Experimental results

Research questions

• RQ1 Can a unified latent-variable framework capture multimodal, interactive futures for variable numbers of agents without labeled modes?
• RQ2 How can discrete latent variables be used to learn meaningful future modes that persist over several timesteps?
• RQ3 Can we perform efficient, closed-form evaluation of conditional trajectory probabilities and hypothetical rollouts for planning?
• RQ4 Does dynamic attention encoding improve state representation of interactions and scene context for multi-agent prediction?

Key findings

• MFP learns semantically meaningful latent modes without supervision, representing distinct behaviors (e.g., straight, stop, right turn) across agents.
• The model scales to arbitrary numbers of agents via shared-parameter per-agent RNNs and a factorized likelihood over time and agents.
• MFP achieves state-of-the-art or competitive results on CARLA, NGSIM, and Argoverse datasets across multiple metrics.
• Hypothetical rollouts allow conditioning on one or more agents’ trajectories to obtain conditional predictions for planning tasks.
• Dynamic attention encoding improves predictive quality by effectively encoding inter-agent relations and scene context.