[Paper Review] Targeted Adversarial Attacks against Neural Network Trajectory Predictors

Kaiyuan Tan, Jun Wang+1 more|arXiv (Cornell University)|Dec 8, 2022

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper proposes TA4TP, a targeted adversarial attack that generates natural-looking perturbations to input trajectories to force DNN-based trajectory predictors to output user-specified target trajectories. By formulating the attack as a constrained nonlinear optimization problem with weighted loss and physical constraints, TA4TP achieves high success rates on state-of-the-art models like Trajectron++ and Grip++ across Apolloscape and nuScenes datasets.

ABSTRACT

Trajectory prediction is an integral component of modern autonomous systems as it allows for envisioning future intentions of nearby moving agents. Due to the lack of other agents' dynamics and control policies, deep neural network (DNN) models are often employed for trajectory forecasting tasks. Although there exists an extensive literature on improving the accuracy of these models, there is a very limited number of works studying their robustness against adversarially crafted input trajectories. To bridge this gap, in this paper, we propose a targeted adversarial attack against DNN models for trajectory forecasting tasks. We call the proposed attack TA4TP for Targeted adversarial Attack for Trajectory Prediction. Our approach generates adversarial input trajectories that are capable of fooling DNN models into predicting user-specified target/desired trajectories. Our attack relies on solving a nonlinear constrained optimization problem where the objective function captures the deviation of the predicted trajectory from a target one while the constraints model physical requirements that the adversarial input should satisfy. The latter ensures that the inputs look natural and they are safe to execute (e.g., they are close to nominal inputs and away from obstacles). We demonstrate the effectiveness of TA4TP on two state-of-the-art DNN models and two datasets. To the best of our knowledge, we propose the first targeted adversarial attack against DNN models used for trajectory forecasting.

Motivation & Objective

• To address the lack of robustness evaluation in DNN-based trajectory forecasting models against adversarial inputs.
• To develop a targeted attack that manipulates DNN predictions to match user-defined desired trajectories.
• To ensure adversarial inputs remain physically plausible and safe by incorporating constraints on trajectory smoothness, lane-keeping, and obstacle avoidance.
• To evaluate the robustness of DNN models not only to adversarial inputs but also to random noise in both clean and adversarial settings.
• To investigate the influence of traffic density on the success rate of adversarial attacks on trajectory prediction models.

Proposed method

• Formulate the attack as a constrained nonlinear optimization problem minimizing a weighted ℓ₂ distance between predicted and target trajectories.
• Assign dynamic weights to states in the target trajectory to prioritize critical points (e.g., final position in a restricted zone).
• Enforce physical constraints such as proximity to the nominal trajectory, lane-keeping, and obstacle avoidance to ensure realism and safety.
• Solve the optimization using gradient-based methods (e.g., Adam optimizer) under full white-box access to the DNN model.
• Integrate trajectory interaction modeling via graph-based attention mechanisms used in the target DNNs (e.g., Trajectron++, Grip++) to preserve context-aware behavior.
• Validate the attack on two SOTA models (Trajectron++, Grip++) using two benchmark datasets (Apolloscape, nuScenes) with diverse traffic scenarios.

Experimental results

Research questions

• RQ1 Can targeted adversarial perturbations be generated to force DNN-based trajectory predictors to output a specific, user-defined trajectory?
• RQ2 How effective is the proposed attack in maintaining physical plausibility and safety of adversarial inputs (e.g., staying within lanes, avoiding obstacles)?
• RQ3 How do DNN models perform under random noise applied to both clean and adversarially crafted inputs, and what does this imply for detection?
• RQ4 Does traffic density affect the success rate of targeted adversarial attacks on trajectory prediction models?
• RQ5 Can the attack be robust to natural perturbations (e.g., wind, road slipperiness), suggesting potential for real-world deployment?

Key findings

• TA4TP successfully forces Trajectron++ and Grip++ to predict target trajectories with average deviations of 0.215 m and 0.331 m, respectively, in high-density traffic scenarios.
• On the Apolloscape dataset, TA4TP achieved a mean deviation of 0.203 m for Grip++ and 0.507 m for Trajectron++ under adversarial inputs with added random noise.
• For the nuScenes dataset, the deviation was 0.253 m (Grip++) and 0.099 m (Trajectron++) under the same noise conditions, indicating robustness to perturbations.
• The DNN models showed higher robustness to random noise in the vicinity of adversarial inputs than near clean inputs, suggesting a potential detection signal.
• The attack remains effective even when most surrounding agents are removed from the scene, indicating resilience to reduced interaction modeling.
• Lower deviation in high-density settings (e.g., 0.034 m for Trajectron++ with all agents) suggests that interaction modeling may enhance attack success, though results vary by model.