[Paper Review] DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets

Junru Gu, Chen Sun+1 more|arXiv (Cornell University)|Aug 22, 2021

Autonomous Vehicle Technology and Safety39 references18 citations

TL;DR

DenseTNT proposes an anchor-free, end-to-end trajectory prediction model that generates diverse, high-coverage future trajectories by predicting dense goal probabilities and selecting optimal goal sets via optimization-based pseudo-labeling. It achieves state-of-the-art performance, ranking 1st on the Argoverse benchmark and winning the 2021 Waymo Open Dataset Motion Prediction Challenge.

ABSTRACT

Due to the stochasticity of human behaviors, predicting the future trajectories of road agents is challenging for autonomous driving. Recently, goal-based multi-trajectory prediction methods are proved to be effective, where they first score over-sampled goal candidates and then select a final set from them. However, these methods usually involve goal predictions based on sparse pre-defined anchors and heuristic goal selection algorithms. In this work, we propose an anchor-free and end-to-end trajectory prediction model, named DenseTNT, that directly outputs a set of trajectories from dense goal candidates. In addition, we introduce an offline optimization-based technique to provide multi-future pseudo-labels for our final online model. Experiments show that DenseTNT achieves state-of-the-art performance, ranking 1st on the Argoverse motion forecasting benchmark and being the 1st place winner of the 2021 Waymo Open Dataset Motion Prediction Challenge.

Motivation & Objective

• To address the limitations of sparse anchor-based goal prediction in multi-trajectory forecasting, such as poor modeling of fine-grained spatial details and reliance on heuristic post-processing like NMS.
• To eliminate the need for rule-based goal selection (e.g., NMS) by directly predicting a set of goals in an end-to-end manner.
• To enable effective training of multi-trajectory prediction models despite the scarcity of multi-future ground truth data by introducing an offline optimization-based pseudo-labeling strategy.
• To improve trajectory prediction performance by leveraging dense goal candidate sampling that captures local spatial variations on lanes.
• To achieve state-of-the-art performance on major benchmarks, including Argoverse and Waymo Open Dataset, through a novel training paradigm and architecture.

Proposed method

• The model uses a sparse context encoder to extract features from HD maps and surrounding agents.
• A dense goal encoder predicts a probability distribution over a large set of densely sampled goal candidates on lane centerlines.
• A goal set predictor performs multi-label prediction to select a final set of high-probability goals from the dense distribution, replacing heuristic post-processing.
• An offline optimization model generates multi-future pseudo-labels by finding the globally optimal goal set from the dense probability map, using the same objective as the online model.
• The online model is trained using these pseudo-labels, enabling end-to-end learning without requiring multiple ground-truth futures per sample.
• The optimization algorithm is run offline to generate high-quality training targets, with runtime tuned to balance performance and efficiency.

Experimental results

Research questions

• RQ1 Can an end-to-end trajectory prediction model achieve superior performance by eliminating sparse anchor-based goal prediction and heuristic post-processing?
• RQ2 How effective is an optimization-based offline model in generating high-quality, diverse pseudo-labels for training an online trajectory predictor?
• RQ3 Does dense goal candidate sampling improve modeling of fine-grained spatial variations compared to sparse anchor methods?
• RQ4 To what extent does the goal set predictor’s design enable the online model to match the performance of the offline optimization model?
• RQ5 What is the impact of goal sampling density and optimization time on the final prediction performance?

Key findings

• DenseTNT achieved the best performance on the Argoverse motion forecasting benchmark, with an mADE of 1.0387 and an mFDE of 1.5514.
• It ranked 1st in the 2021 Waymo Open Dataset Motion Prediction Challenge, achieving an mAP of 0.3281, outperforming all other methods.
• The model’s performance closely matched that of the offline optimization model, demonstrating the effectiveness of the goal set predictor and the training paradigm.
• Increasing goal sampling density from 3.0m to 0.5m reduced minFDE from 1.42 to 1.27, with no further improvement beyond 1.0m, indicating saturation at high density.
• Optimization time per sample had a significant impact: performance improved rapidly within 100ms and stabilized after 200ms, with no further gains beyond 500ms.
• The dense probability estimation outperformed sparse estimation, especially when combined with the goal set predictor, and significantly surpassed NMS-based baselines.