[Paper Review] Fooling Detection Alone is Not Enough: First Adversarial Attack against Multiple Object Tracking

Yunhan Jia, Yantao Lu+4 more|arXiv (Cornell University)|May 27, 2019

Adversarial Robustness in Machine Learning31 references19 citations

TL;DR

This paper introduces tracker hijacking, a novel adversarial attack that targets the multiple object tracking (MOT) stage in autonomous driving by exploiting tracking error reduction, enabling successful attacks with as few as one frame of adversarial perturbations. It achieves nearly 100% success rate on the Berkeley Deep Drive dataset when 3 frames are attacked, far surpassing detection-only attacks that max out at 25% under the same conditions.

ABSTRACT

Recent work in adversarial machine learning started to focus on the visual perception in autonomous driving and studied Adversarial Examples (AEs) for object detection models. However, in such visual perception pipeline the detected objects must also be tracked, in a process called Multiple Object Tracking (MOT), to build the moving trajectories of surrounding obstacles. Since MOT is designed to be robust against errors in object detection, it poses a general challenge to existing attack techniques that blindly target objection detection: we find that a success rate of over 98% is needed for them to actually affect the tracking results, a requirement that no existing attack technique can satisfy. In this paper, we are the first to study adversarial machine learning attacks against the complete visual perception pipeline in autonomous driving, and discover a novel attack technique, tracker hijacking, that can effectively fool MOT using AEs on object detection. Using our technique, successful AEs on as few as one single frame can move an existing object in to or out of the headway of an autonomous vehicle to cause potential safety hazards. We perform evaluation using the Berkeley Deep Drive dataset and find that on average when 3 frames are attacked, our attack can have a nearly 100% success rate while attacks that blindly target object detection only have up to 25%.

Motivation & Objective

• To address the gap in adversarial machine learning research that focuses only on object detection while neglecting the robustness of multiple object tracking (MOT) in autonomous driving.
• To investigate whether adversarial attacks on object detection alone can effectively compromise the final tracking output, given MOT's inherent error correction mechanisms.
• To develop a new attack technique that exploits MOT's tracking error reduction process to mislead object trajectories with minimal adversarial perturbations.
• To evaluate the effectiveness of this new attack under realistic conditions using the Berkeley Deep Drive dataset and standard MOT configurations.
• To advocate for MOT accuracy as a critical evaluation metric in future adversarial machine learning research for autonomous systems.

Proposed method

• The attack generates adversarial bounding boxes that target the tracking error reduction process in MOT, specifically by manipulating the association between detections and existing tracks.
• It leverages the YOLOv3 object detector to generate adversarial examples that are designed to mislead the tracker into associating detections with incorrect trajectories.
• The adversarial perturbations are applied to a small number of frames—often just one or two—targeting specific object locations to induce trajectory deviation.
• The method exploits the fact that MOT uses consistency and stability across frames to maintain tracks, and by introducing subtle adversarial shifts, it can hijack the predicted trajectory.
• The attack is evaluated using the Berkeley Deep Drive dataset under realistic noise levels and standard MOT algorithm configurations (e.g., IoU-based data association).
• The approach is generalizable to other detection models and tracking mechanisms, including appearance-based matching, with minor adaptations.

Experimental results

Research questions

• RQ1 Can adversarial attacks on object detection alone effectively compromise the final tracking output in autonomous driving systems?
• RQ2 Why do existing detection-only attacks fail to influence MOT despite high success rates on detection?
• RQ3 How can adversarial perturbations be strategically placed to exploit MOT’s error reduction mechanism and hijack object trajectories with minimal frames?
• RQ4 What is the minimum number of frames needed for an adversarial attack to successfully alter the predicted trajectory of a tracked object in MOT?
• RQ5 How does tracker hijacking compare quantitatively to detection-only attacks in terms of success rate under identical conditions?

Key findings

• Tracker hijacking achieves a nearly 100% success rate when 3 consecutive frames are attacked, significantly outperforming detection-only attacks that cap at 25% under the same conditions.
• The attack can successfully alter object trajectories with as few as one adversarial frame, demonstrating high efficiency and low detectability.
• Even with a detection success rate of only 98.3%, detection-only attacks fail to reliably influence MOT, as they require 60 consecutive successful frames at 30 fps to maintain track disruption.
• The tracker hijacking attack is robust under realistic noise levels and standard MOT configurations, confirming its practical feasibility.
• The method is generalizable to other object detection models and tracking mechanisms, such as appearance-based association, with minimal adaptation.
• The study reveals that MOT is a critical but overlooked component in adversarial machine learning research for autonomous driving, necessitating its inclusion in future evaluation frameworks.