[Paper Review] Sliding Sequential CVAE with Time Variant Socially-aware Rethinking for Trajectory Prediction

Hao Zhou, Dongchun Ren+3 more|arXiv (Cornell University)|Oct 28, 2021

Autonomous Vehicle Technology and Safety33 references4 citations

TL;DR

This paper proposes CSR, a cascaded conditional variational autoencoder with socially-aware rethinking for pedestrian trajectory prediction, which sequentially predicts future points using updated past trajectories and refines them via social interaction modeling. The method achieves state-of-the-art performance, reducing ADE by 38.0% on SDD and 22.2% on ETH/UCY, with a more efficient slide CVAE variant cutting model parameters by 85.9% and boosting inference speed by 6.3x.

ABSTRACT

Pedestrian trajectory prediction is a key technology in many applications such as video surveillance, social robot navigation, and autonomous driving, and significant progress has been made in this research topic. However, there remain two limitations of previous studies. First, with the continuation of time, the prediction error at each time step increases significantly, causing the final displacement error to be impossible to ignore. Second, the prediction results of multiple pedestrians might be impractical in the prediction horizon, i.e., the predicted trajectories might collide with each other. To overcome these limitations, this work proposes a novel trajectory prediction method called CSR, which consists of a cascaded conditional variational autoencoder (CVAE) module and a socially-aware regression module. The cascaded CVAE module first estimates the future trajectories in a sequential pattern. Specifically, each CVAE concatenates the past trajectories and the predicted points so far as the input and predicts the location at the following time step. Then, the socially-aware regression module generates offsets from the estimated future trajectories to produce the socially compliant final predictions, which are more reasonable and accurate results than the estimated trajectories. Moreover, considering the large model parameters of the cascaded CVAE module, a slide CVAE module is further exploited to improve the model efficiency using one shared CVAE, in a slidable manner. Experiments results demonstrate that the proposed method exhibits improvements over state-of-the-art method on the Stanford Drone Dataset (SDD) and ETH/UCY of approximately 38.0% and 22.2%, respectively.

Motivation & Objective

• To address increasing prediction error over time in long-horizon pedestrian trajectory forecasting.
• To resolve socially implausible trajectory predictions where pedestrians collide or violate interaction norms.
• To improve model efficiency without sacrificing performance in sequential trajectory prediction.
• To develop a method that leverages predicted points as pseudo-labels to reduce uncertainty in subsequent predictions.
• To integrate social interaction modeling into the prediction refinement process for more realistic and compliant trajectories.

Proposed method

• The cascaded CVAE module sequentially predicts future trajectory points by conditioning each prediction on the past trajectory and all previously predicted points, treating them as pseudo-labels to reduce uncertainty.
• Each CVAE in the cascade uses a shared latent space but operates independently, with inputs composed of past trajectories and cumulative predicted points.
• The socially-aware regression module refines the predicted trajectories by extracting interaction features from both past and predicted trajectories, enforcing social compliance.
• A slide CVAE module is introduced to replace unshared CVAEs, using a single shared CVAE in a slidable window fashion to drastically reduce model parameters and improve inference speed.
• The model is trained end-to-end with a reconstruction loss for the CVAE and a regression loss for the socially-aware refinement module.
• The method incorporates time-variant social rethinking by dynamically modeling interactions based on evolving predicted positions, enhancing realism in complex scenarios.

Experimental results

Research questions

• RQ1 Can sequential prediction using updated past trajectories and predicted points reduce long-term prediction error in pedestrian trajectory forecasting?
• RQ2 How can social interactions between pedestrians be effectively modeled to produce collision-free and realistic trajectory predictions?
• RQ3 Can a shared CVAE architecture in a sliding window mechanism maintain performance while significantly reducing model parameters and improving inference speed?
• RQ4 To what extent does refining predicted trajectories with social interaction features improve prediction accuracy and realism?
• RQ5 How does the proposed method compare to state-of-the-art approaches in terms of ADE and FDE across diverse pedestrian interaction scenarios?

Key findings

• The proposed CSR method achieves a 38.0% relative improvement in ADE over the state-of-the-art on the Stanford Drone Dataset (SDD).
• On the ETH/UCY benchmark, CSR reduces ADE by 22.2% compared to the best existing method.
• The socially-aware regression module improves performance by 17.6% on ETH/UCY and 20.4% on SDD in terms of ADE, demonstrating its effectiveness in refining socially compliant trajectories.
• The slide CVAE variant reduces model parameters by 85.9% and increases inference speed by 6.3x compared to the cascaded CVAE, while improving ADE by 43.1% on SDD.
• Qualitative results show that CSR generates more realistic and collision-free trajectories than PECNet, especially in complex interaction scenarios like crossing and collision avoidance.
• The method outperforms existing SOTA models across all evaluation metrics, establishing a new state-of-the-art performance in pedestrian trajectory prediction.