[Paper Review] Segment and Complete: Defending Object Detectors against Adversarial Patch Attacks with Robust Patch Detection

Jiang Liu, Alexander Levine+3 more|arXiv (Cornell University)|Dec 8, 2021

Adversarial Robustness in Machine Learning6 citations

TL;DR

This paper proposes Segment and Complete (SAC), a defense framework that detects and removes adversarial patches from object detection inputs via robust patch segmentation and shape completion. SAC achieves 45.0% mAP under 100×100 patch attacks—30.6% higher than the undefended model—while maintaining 49.0% mAP on clean images, and generalizes well across unseen patch shapes, attack budgets, and methods.

ABSTRACT

Object detection plays a key role in many security-critical systems. Adversarial patch attacks, which are easy to implement in the physical world, pose a serious threat to state-of-the-art object detectors. Developing reliable defenses for object detectors against patch attacks is critical but severely understudied. In this paper, we propose Segment and Complete defense (SAC), a general framework for defending object detectors against patch attacks through detection and removal of adversarial patches. We first train a patch segmenter that outputs patch masks which provide pixel-level localization of adversarial patches. We then propose a self adversarial training algorithm to robustify the patch segmenter. In addition, we design a robust shape completion algorithm, which is guaranteed to remove the entire patch from the images if the outputs of the patch segmenter are within a certain Hamming distance of the ground-truth patch masks. Our experiments on COCO and xView datasets demonstrate that SAC achieves superior robustness even under strong adaptive attacks with no reduction in performance on clean images, and generalizes well to unseen patch shapes, attack budgets, and unseen attack methods. Furthermore, we present the APRICOT-Mask dataset, which augments the APRICOT dataset with pixel-level annotations of adversarial patches. We show SAC can significantly reduce the targeted attack success rate of physical patch attacks. Our code is available at https://github.com/joellliu/SegmentAndComplete.

Motivation & Objective

• Address the critical lack of robust defenses for object detectors against physical and digital adversarial patch attacks.
• Overcome the challenge of localized adversarial patches that disrupt object detection globally due to large receptive fields in deep models.
• Develop a general-purpose defense that does not require retraining the base object detector.
• Enable pixel-level patch detection and reliable removal even under adaptive attacks.
• Create a new benchmark dataset with pixel-level annotations for physical adversarial patches to support future research.

Proposed method

• Train a patch segmenter using pixel-level annotations from the newly introduced APRICOT-Mask dataset to generate initial adversarial patch masks.
• Apply self-adversarial training to enhance the robustness of the patch segmenter against adversarial perturbations.
• Design a robust shape completion algorithm that guarantees full patch coverage if the initial mask is within a specified Hamming distance of the ground truth.
• Use the completed mask to black out the adversarial patch region in the input image before feeding it to the base object detector.
• Ensure the defense is object-detector-agnostic by decoupling patch detection from detection inference.
• Leverage spatial context and shape priors to improve detection reliability even when initial segmentations are imperfect.

Experimental results

Research questions

• RQ1 Can a pixel-level patch detection method significantly improve robustness of object detectors against adversarial patch attacks?
• RQ2 How effective is self-adversarial training in improving the robustness of a patch segmenter under adaptive attacks?
• RQ3 Can a shape completion algorithm guarantee complete patch removal when the initial detection is partially accurate?
• RQ4 To what extent does the defense generalize to unseen patch shapes, attack budgets, and attack methods?
• RQ5 How does the proposed defense perform on real-world physical patch attacks compared to existing defenses?

Key findings

• SAC achieves 45.0% mAP on COCO under 100×100 adversarial patch attacks, representing a 30.6% mAP gain over the undefended model.
• The defense maintains 49.0% mAP on clean COCO images, demonstrating no performance degradation on clean data.
• On the xView dataset, SAC reduces the targeted physical patch attack success rate from 7.97% to 2.17%.
• SAC generalizes effectively to unseen patch shapes (e.g., circle, ellipse, rectangle), even when the shape completion prior assumes square patches.
• Failure cases are rare and primarily stem from undetected patches or masking-induced false positives, which can be mitigated via model fine-tuning or content inpainting.
• The APRICOT-Mask dataset provides the first publicly available pixel-level annotations for physical adversarial patches, enabling future research in patch detection and removal.