[Paper Review] Diverse Trajectory Forecasting with Determinantal Point Processes

Ye Yuan, Kris Kitani|arXiv (Cornell University)|Jul 11, 2019

Point processes and geometric inequalitiesMathematics46 references67 citations

TL;DR

The paper introduces a diversity sampling function (DSF) that, guided by a DPP-based diversity loss, produces a diverse yet likely set of future trajectories decoded from a cVAE, improving multi-modal trajectory forecasting.

ABSTRACT

The ability to forecast a set of likely yet diverse possible future behaviors of an agent (e.g., future trajectories of a pedestrian) is essential for safety-critical perception systems (e.g., autonomous vehicles). In particular, a set of possible future behaviors generated by the system must be diverse to account for all possible outcomes in order to take necessary safety precautions. It is not sufficient to maintain a set of the most likely future outcomes because the set may only contain perturbations of a single outcome. While generative models such as variational autoencoders (VAEs) have been shown to be a powerful tool for learning a distribution over future trajectories, randomly drawn samples from the learned implicit likelihood model may not be diverse -- the likelihood model is derived from the training data distribution and the samples will concentrate around the major mode that has most data. In this work, we propose to learn a diversity sampling function (DSF) that generates a diverse and likely set of future trajectories. The DSF maps forecasting context features to a set of latent codes which can be decoded by a generative model (e.g., VAE) into a set of diverse trajectory samples. Concretely, the process of identifying the diverse set of samples is posed as a parameter estimation of the DSF. To learn the parameters of the DSF, the diversity of the trajectory samples is evaluated by a diversity loss based on a determinantal point process (DPP). Gradient descent is performed over the DSF parameters, which in turn move the latent codes of the sample set to find an optimal diverse and likely set of trajectories. Our method is a novel application of DPPs to optimize a set of items (trajectories) in continuous space. We demonstrate the diversity of the trajectories produced by our approach on both low-dimensional 2D trajectory data and high-dimensional human motion data.

Motivation & Objective

• Motivate the need for diverse future trajectory sets in safety-critical perception systems.
• Propose a DSF that maps forecasting context to latent codes decoded by a cVAE to generate diverse samples.
• Leverage a determinantal point process (DPP) diversity loss to optimize DSF parameters via gradient descent.
• Show that DSF yields more diverse and representative trajectory sets than baselines across low- and high-dimensional data.

Proposed method

• Train a conditional VAE (cVAE) to model p(x|ψ) for future trajectories.
• Introduce DSF γ-parameterized neural network that outputs a set of latent codes z for decoding into trajectories.
• Define a DPP-based diversity loss using L = Diag(r) S Diag(r) with S as a Gaussian-style similarity over trajectories and r as a latent-space quality vector.
• Compute the diversity objective as L_diverse(γ) = -tr(I - (L(γ) + I)^{-1}) and optimize γ via gradient descent.
• At inference, generate a DSF ground set and (optionally) perform MAP inference on the DPP to choose a diverse subset.
• Use N samples (budget) from the DSF-derived latent codes and decode with the cVAE decoder to obtain Y.

Experimental results

Research questions

• RQ1 Can a learned DSF produce a more diverse and representative set of future trajectories than standard sampling from a cVAE?
• RQ2 How does DSF perform under balanced vs. imbalanced data, and across low- and high-dimensional trajectory forecasting tasks?
• RQ3 Does the DPP-based diversity objective stabilize training and improve diversity without sacrificing trajectory quality?

Key findings

• DSF consistently outperforms baselines (cVAE, MCL, R2P2, cGAN) in diversity-aware metrics on synthetic data under balanced and imbalanced settings.
• On synthetic 2D data (N=10), DSF achieves ADE 0.182 and 0.198, FDE 0.344 and 0.371 for balanced/imbalanced respectively, with higher ASD and FSD indicating less repetition.
• On human motion data, DSF yields lower ADE/FDE (e.g., 0.259/0.421 for N=10) and higher diversity (ASD 0.115, FSD 0.282) than baselines.
• In large-scale Human3.6M experiments (N=10 and N=50), DSF variants show favorable trade-offs between diversity and quality, with DSF achieving competitive ADE/FDE and notably higher diversity metrics in several configurations.