[Paper Review] View Vertically: A Hierarchical Network for Trajectory Prediction via Fourier Spectrums

Conghao Wong, Beihao Xia+6 more|arXiv (Cornell University)|Oct 14, 2021

Autonomous Vehicle Technology and Safety26 references4 citations

TL;DR

This paper proposes V²-Net, a hierarchical trajectory prediction model that leverages Discrete Fourier Transforms (DFT) to analyze agents' motion preferences across frequency bands—low frequencies for coarse motion trends and high frequencies for fine-grained interactions. The method uses a two-stage approach: first estimating key spectral components at critical frequencies, then interpolating to reconstruct smooth, plausible trajectories, achieving state-of-the-art performance on ETH-UCY and Stanford Drone datasets.

ABSTRACT

Understanding and forecasting future trajectories of agents are critical for behavior analysis, robot navigation, autonomous cars, and other related applications. Previous methods mostly treat trajectory prediction as time sequence generation. Different from them, this work studies agents' trajectories in a "vertical" view, i.e., modeling and forecasting trajectories from the spectral domain. Different frequency bands in the trajectory spectrums could hierarchically reflect agents' motion preferences at different scales. The low-frequency and high-frequency portions could represent their coarse motion trends and fine motion variations, respectively. Accordingly, we propose a hierarchical network V$^2$-Net, which contains two sub-networks, to hierarchically model and predict agents' trajectories with trajectory spectrums. The coarse-level keypoints estimation sub-network first predicts the "minimal" spectrums of agents' trajectories on several "key" frequency portions. Then the fine-level spectrum interpolation sub-network interpolates the spectrums to reconstruct the final predictions. Experimental results display the competitiveness and superiority of V$^2$-Net on both ETH-UCY benchmark and the Stanford Drone Dataset.

Motivation & Objective

• To address the limitation of existing methods that treat trajectory prediction as sequential generation, missing hierarchical motion preferences across temporal scales.
• To explore a 'vertical' spectral view of trajectories using Fourier transforms to capture coarse and fine motion patterns simultaneously.
• To develop a hierarchical deep learning framework that models global plannings (low-frequency components) and interactive preferences (high-frequency components) separately.
• To improve prediction accuracy and plausibility by incorporating scene constraints and physical rules through spectral reconstruction.
• To enable controllable, multi-modal trajectory generation by adjusting the number of spectral keypoints.

Proposed method

• The model employs a two-branch architecture: a coarse-level keypoints estimation sub-network that predicts spectral components at selected key frequencies.
• The fine-level spectrum interpolation sub-network reconstructs full trajectories by interpolating between the estimated spectral keypoints using inverse DFT.
• A latent noise vector $ z \sim N(0,I) $ is used in the keypoint sub-network to generate multiple stochastic predictions, enabling multi-modal outputs.
• The method leverages the Discrete Fourier Transform (DFT) to convert 2D trajectory sequences into spectral representations, separating low- and high-frequency components.
• Trajectories are reconstructed by superimposing sinusoidal components from the interpolated spectrum, ensuring spatio-temporal continuity and smoothness.
• The model is trained with a combination of reconstruction loss and adversarial loss to improve realism and diversity of predicted trajectories.

Experimental results

Research questions

• RQ1 Can modeling trajectories in the spectral domain reveal hierarchical motion patterns that are not easily observable in the time domain?
• RQ2 How can low-frequency and high-frequency components of trajectory spectrums be used to model coarse motion trends and fine interaction details separately?
• RQ3 To what extent can a two-stage hierarchical framework using spectral keypoints and interpolation improve trajectory prediction accuracy and diversity?
• RQ4 How does the number of spectral keypoints affect the trade-off between prediction diversity and plausibility?
• RQ5 Can spectral-based modeling better capture physical constraints and social interaction rules than standard recurrent sequence modeling?

Key findings

• V²-Net achieves state-of-the-art performance on both the ETH-UCY benchmark and the Stanford Drone Dataset, demonstrating superior prediction accuracy.
• Increasing the number of spectral keypoints ($ N_{key} $) improves prediction plausibility by better constraining motion trends, but too many keypoints reduce diversity and degrade performance.
• The model generates smoother, more physically realistic trajectories due to the inherent continuity of sinusoidal reconstruction from DFT components.
• Visualizations show that V²-Net respects scene constraints—e.g., avoiding grass or navigating traffic circles correctly—indicating strong generalization to complex environments.
• The method successfully captures multi-modal behaviors, such as multiple valid paths through intersections or traffic circles, with contextually appropriate choices.
• Ablation studies confirm that the hierarchical spectral approach outperforms standard sequence-based models in both quantitative metrics and qualitative realism.