[Paper Review] Feasibility of Inconspicuous GAN-generated Adversarial Patches against Object Detection

Svetlana Pavlitskaya, Bianca-Marina Codău+1 more|arXiv (Cornell University)|Jul 15, 2022

Generative Adversarial Networks and Image Synthesis6 citations

TL;DR

This paper evaluates GAN-based methods for generating inconspicuous adversarial patches that evade human detection while effectively attacking object detectors. By adapting pretrained GANs (DCGAN, BigGAN) and combining them with PGD optimization, the authors achieve realistic-looking patches with strong attack performance, particularly showing that pretrained GANs yield superior visual fidelity without sacrificing attack efficacy on YOLOv3 using COCO dataset.

ABSTRACT

Standard approaches for adversarial patch generation lead to noisy conspicuous patterns, which are easily recognizable by humans. Recent research has proposed several approaches to generate naturalistic patches using generative adversarial networks (GANs), yet only a few of them were evaluated on the object detection use case. Moreover, the state of the art mostly focuses on suppressing a single large bounding box in input by overlapping it with the patch directly. Suppressing objects near the patch is a different, more complex task. In this work, we have evaluated the existing approaches to generate inconspicuous patches. We have adapted methods, originally developed for different computer vision tasks, to the object detection use case with YOLOv3 and the COCO dataset. We have evaluated two approaches to generate naturalistic patches: by incorporating patch generation into the GAN training process and by using the pretrained GAN. For both cases, we have assessed a trade-off between performance and naturalistic patch appearance. Our experiments have shown, that using a pre-trained GAN helps to gain realistic-looking patches while preserving the performance similar to conventional adversarial patches.

Motivation & Objective

• To investigate the feasibility of generating naturalistic, inconspicuous adversarial patches that evade human detection while attacking object detectors.
• To address the limitation of existing methods that focus only on directly overlapping large objects, by targeting objects near the patch location.
• To evaluate the trade-off between visual realism of patches and attack effectiveness in digital, per-instance and universal settings.
• To compare two GAN-based approaches: end-to-end PGD-GAN training and using pretrained GANs for patch generation.
• To assess performance on YOLOv3 using the COCO dataset, focusing on mAP degradation and patch visual quality.

Proposed method

• Adapted GAN-based patch generation methods—specifically, combining PGD optimization with GAN training (PGD-GAN) and using pretrained GANs (DCGAN, BigGAN)—for object detection attacks.
• Applied PGD-based adversarial training to optimize patches that minimize object detector confidence scores for nearby objects, not just directly overlapped ones.
• Used a combination of loss functions: adversarial loss, non-printability loss, and total variation loss to ensure patch realism and printability.
• Evaluated both per-instance and universal attack settings, where a single patch is applied across multiple images.
• Generated patches using latent space manipulation (e.g., latent shift, brightness/contrast transformations) on pretrained GANs to improve visual quality.
• Trained and evaluated on the COCO dataset using YOLOv3 as the target detector, measuring attack success via mean average precision (mAP) degradation.

Experimental results

Research questions

• RQ1 Can GAN-based methods generate adversarial patches that are visually realistic and inconspicuous while maintaining strong attack performance on object detectors?
• RQ2 How does the use of pretrained GANs compare to end-to-end PGD-GAN training in terms of patch realism and attack success?
• RQ3 What is the trade-off between visual fidelity of the generated patch and the resulting mAP degradation in object detection?
• RQ4 Does attacking objects near the patch (rather than directly overlapping) remain effective with inconspicuous patches?
• RQ5 How do different GAN architectures (DCGAN vs. BigGAN) influence the realism and effectiveness of the adversarial patches?

Key findings

• Pretrained GANs (especially BigGAN) produced patches with significantly higher visual realism compared to PGD-GAN or conventional methods, as confirmed by qualitative analysis.
• The pretrained DCGAN with brightness and contrast transformations achieved the best balance between attack performance and visual quality, outperforming the PGD-GAN approach in mAP degradation.
• The patch generated using Hu et al.'s method (2021) achieved comparable mAP to the pretrained DCGAN with transformations but had worse performance on the 'car' class and was less realistic than BigGAN-based patches.
• The pretrained BigGAN patch with transformations showed the highest visual realism but the weakest attack performance, with mAP as low as 4.4%—worse than the baseline black square attack.
• All GAN-based approaches reduced mAP significantly compared to clean images, demonstrating effective attack capability despite not matching the performance of conspicuous adversarial patches.
• The study confirmed a clear trade-off: higher visual realism (via BigGAN) correlated with lower attack success, while more effective attacks (via DCGAN) came at the cost of reduced visual fidelity.