[Paper Review] Fooling Detection Alone is Not Enough: Adversarial Attack against Multiple Object Tracking

Yunhan Jia, Yantao Lu+5 more|arXiv (Cornell University)|Apr 30, 2020

Adversarial Robustness in Machine Learning32 references56 citations

TL;DR

This paper introduces tracker hijacking, a novel adversarial attack that targets the entire visual perception pipeline in autonomous driving by exploiting weaknesses in Multiple Object Tracking (MOT) rather than just object detection. By crafting adversarial examples on as few as one frame, the attack achieves nearly 100% success in manipulating object trajectories—demonstrating that fooling detection alone is insufficient, as MOT's robustness demands over 98% attack success on detection to impact tracking.

ABSTRACT

Recent work in adversarial machine learning started to focus on the visual perception in autonomous driving and studied Adversarial Examples (AEs) for object detection models. However, in such visual perception pipeline the detected objects must also be tracked, in a process called Multiple Object Tracking (MOT), to build the moving trajectories of surrounding obstacles. Since MOT is designed to be robust against errors in object detection, it poses a general challenge to existing attack techniques that blindly target objection detection: we find that a success rate of over 98% is needed for them to actually affect the tracking results, a requirement that no existing attack technique can satisfy. In this paper, we are the first to study adversarial machine learning attacks against the complete visual perception pipeline in autonomous driving, and discover a novel attack technique, tracker hijacking, that can effectively fool MOT using AEs on object detection. Using our technique, successful AEs on as few as one single frame can move an existing object in to or out of the headway of an autonomous vehicle to cause potential safety hazards. We perform evaluation using the Berkeley Deep Drive dataset and find that on average when 3 frames are attacked, our attack can have a nearly 100% success rate while attacks that blindly target object detection only have up to 25%.

Motivation & Objective

• To address the gap in adversarial attacks that focus only on object detection while ignoring the robustness of Multiple Object Tracking (MOT) in autonomous driving pipelines.
• To investigate why existing attacks fail to disrupt MOT despite high success on object detection.
• To develop a new attack strategy that effectively compromises MOT by exploiting its tracking mechanisms.
• To evaluate the real-world safety implications of adversarial attacks on the complete perception pipeline, including trajectory manipulation.

Proposed method

• Design a novel attack technique called 'tracker hijacking' that specifically targets the tracking component of MOT, rather than just the detection stage.
• Generate adversarial examples that are crafted to be effective even under the high success rate threshold (over 98%) required to disrupt MOT.
• Use the Berkeley Deep Drive dataset to simulate realistic autonomous driving scenarios and evaluate attack effectiveness across multiple frames.
• Optimize adversarial perturbations to minimize the number of frames needed to achieve tracking failure, focusing on single-frame attacks.
• Analyze the interaction between object detection and tracking modules to identify vulnerabilities in MOT's response to adversarial inputs.
• Compare the performance of tracker hijacking against traditional detection-only attacks under identical conditions.

Experimental results

Research questions

• RQ1 Why do existing adversarial attacks fail to disrupt Multiple Object Tracking (MOT) despite high success on object detection?
• RQ2 What is the minimum attack success rate required on object detection to successfully manipulate MOT outcomes?
• RQ3 Can adversarial examples be designed to directly exploit tracking mechanisms and cause trajectory manipulation with minimal frame perturbations?
• RQ4 How effective is tracker hijacking compared to detection-only attacks in real-world autonomous driving scenarios?
• RQ5 What are the safety implications of adversarial attacks that manipulate object trajectories in the headway of autonomous vehicles?

Key findings

• When 3 frames are attacked, the proposed tracker hijacking method achieves a nearly 100% success rate in manipulating object trajectories.
• In contrast, traditional detection-only attacks achieve a maximum success rate of only 25% under the same conditions.
• The study reveals that MOT requires over 98% attack success on detection to be effectively disrupted, a threshold unmet by existing techniques.
• A single adversarial frame is sufficient to cause a tracked object to be inserted into or removed from the vehicle's headway, posing direct safety risks.
• The results demonstrate that fooling object detection alone is insufficient for effective adversarial attacks on the full perception pipeline.
• The tracker hijacking method successfully exploits the interaction between detection and tracking, making it significantly more effective than detection-focused attacks.