[Paper Review] Trajectory Forecasting from Detection with Uncertainty-Aware Motion Encoding

Pu Zhang, Lei Bai+4 more|arXiv (Cornell University)|Feb 3, 2022

Anomaly Detection Techniques and Applications4 citations

TL;DR

This paper proposes a trajectory forecasting model that directly uses object detection outputs instead of relying on tracked trajectories, incorporating uncertainty-aware motion encoding via affinity-aware state update and multiple state aggregation to improve robustness against noisy detections. The method achieves state-of-the-art performance on nuScenes, outperforming existing detection-based and joint tracking-forecasting methods by up to 25% in FDE under challenging conditions.

ABSTRACT

Trajectory forecasting is critical for autonomous platforms to make safe planning and actions. Currently, most trajectory forecasting methods assume that object trajectories have been extracted and directly develop trajectory predictors based on the ground truth trajectories. However, this assumption does not hold in practical situations. Trajectories obtained from object detection and tracking are inevitably noisy, which could cause serious forecasting errors to predictors built on ground truth trajectories. In this paper, we propose a trajectory predictor directly based on detection results without relying on explicitly formed trajectories. Different from the traditional methods which encode the motion cue of an agent based on its clearly defined trajectory, we extract the motion information only based on the affinity cues among detection results, in which an affinity-aware state update mechanism is designed to take the uncertainty of association into account. In addition, considering that there could be multiple plausible matching candidates, we aggregate the states of them. This design relaxes the undesirable effect of noisy trajectory obtained from data association. Extensive ablation experiments validate the effectiveness of our method and its generalization ability on different detectors. Cross-comparison to other forecasting schemes further proves the superiority of our method. Code will be released upon acceptance.

Motivation & Objective

• To address the limitations of existing trajectory forecasting methods that rely on ground-truth or tracked trajectories, which are often corrupted by detection and tracking noise.
• To develop a forecasting model that directly consumes detection results without requiring explicitly formed trajectories.
• To model association uncertainty between detections to reduce the impact of noisy or erroneous data associations on motion encoding.
• To enhance motion representation by aggregating features from multiple plausible detection candidates, improving robustness to localization drift and false positives.
• To ensure generalization across different object detectors and maintain performance under real-world detection noise.

Proposed method

• Introduces an affinity-aware state update (ASU) mechanism that uses motion-aware affinity scores between detections as a proxy for association uncertainty, guiding the state update process in motion encoding.
• Employs multiple state aggregation (MSA) to combine features from multiple plausible detection candidates at each time step, reducing sensitivity to single-point errors.
• Designs a motion-aware affinity measuring module that leverages long-term motion cues to compute matching confidence between detections, reflecting uncertainty in data association.
• Constructs a unified framework with three modules: affinity measurement, uncertain-aware motion encoding (using ASU and MSA), and trajectory decoding, all operating directly on detection results.
• Uses implicit trajectories—temporally linked detections without explicit tracking—to represent agent motion, avoiding reliance on potentially erroneous tracked paths.
• Applies a learnable attention mechanism (SIM function) to weight the contribution of different detection candidates in the state representation, enhancing robustness to false positives.

Experimental results

Research questions

• RQ1 Can trajectory forecasting be effectively performed directly from detection results without relying on explicitly tracked trajectories?
• RQ2 How can association uncertainty between detections be modeled and integrated into motion encoding to improve forecasting robustness?
• RQ3 Can multiple plausible detection candidates be effectively aggregated to enhance motion representation and reduce sensitivity to noisy detections?
• RQ4 Does the proposed method generalize across different object detectors and outperform existing detection-based and joint tracking-forecasting approaches?
• RQ5 To what extent does incorporating uncertainty and multiple candidates mitigate the impact of false positives and localization drift in trajectory forecasting?

Key findings

• The proposed method achieves an FDE@3s of 93.4 on the nuScenes dataset when combined with the FreeAnchor detector, outperforming PTP (122.7) and PnPNet (96) under the same evaluation protocol.
• With the CentrNet detector, the method achieves an FDE@3s of 97.7, demonstrating strong generalization across different detection backbones.
• The model reduces forecasting error by up to 25% compared to PTP and shows superior performance even when compared to SOTA joint detection-forecasting models like PnPNet.
• The ASU and MSA mechanisms significantly improve robustness: the model can correctly predict trajectories for false positives by leveraging nearby motion cues, unlike baseline methods that fail on such cases.
• Ablation studies confirm that both ASU and MSA contribute independently to performance, with ASU being especially effective in handling unreliable last-frame detections.
• The method maintains high performance on both Car and Vehicle classes, with FDE@3s of 104.1 and 105.1 respectively under Setting 2, showing consistent superiority across object categories.