[Paper Review] Generating Adversarial yet Inconspicuous Patches with a Single Image

Jinqi Luo, Tao Bai+1 more|arXiv (Cornell University)|Sep 21, 2020

Adversarial Robustness in Machine Learning32 references4 citations

TL;DR

This paper proposes a GAN-based method to generate adversarial patches that are both highly effective and visually inconspicuous using only a single image. By employing a coarse-to-fine, multi-scale generator-discriminator framework with perceptual sensitivity-guided patch placement and contextual consistency modeling, the approach achieves high attack success rates (up to 100%) while evading human detection in 36.96% of cases—significantly lower than existing methods—making it the first one-shot adversarial patch generation method with strong evasion capability.

ABSTRACT

Deep neural networks have been shown vulnerable toadversarial patches, where exotic patterns can resultin models wrong prediction. Nevertheless, existing ap-proaches to adversarial patch generation hardly con-sider the contextual consistency between patches andthe image background, causing such patches to be eas-ily detected and adversarial attacks to fail. On the otherhand, these methods require a large amount of data fortraining, which is computationally expensive. To over-come these challenges, we propose an approach to gen-erate adversarial yet inconspicuous patches with onesingle image. In our approach, adversarial patches areproduced in a coarse-to-fine way with multiple scalesof generators and discriminators. Contextual informa-tion is encoded during the Min-Max training to makepatches consistent with surroundings. The selection ofpatch location is based on the perceptual sensitivity ofvictim models. Through extensive experiments, our ap-proach shows strong attacking ability in both the white-box and black-box setting. Experiments on saliency de-tection and user evaluation indicate that our adversar-ial patches can evade human observations, demonstratethe inconspicuousness of our approach. Lastly, we showthat our approach preserves the attack ability in thephysical world.

Motivation & Objective

• To address the limitations of existing adversarial patch methods that require large-scale training data and produce visually conspicuous patches.
• To develop a method that generates adversarial patches from a single image, reducing data and computational demands.
• To improve patch inconspicuousness by ensuring contextual consistency with the image background.
• To enhance attack success and transferability across diverse victim models while minimizing human detectability.

Proposed method

• A coarse-to-fine multi-scale GAN framework with multiple generator-discriminator pairs trained on image pyramids at different resolutions.
• Patch generation is guided by a victim model’s attention map to target the most sensitive regions for higher attack success.
• Discriminators are conditioned on both the generated patch and its surrounding context, ensuring spatial and contextual consistency.
• The generator uses skip connections from coarser-scale patches (upsampled) as input to maintain structural coherence across scales.
• The training objective combines adversarial loss, attack loss (fooling the victim model), reconstruction loss, and total variation loss for smooth textures.
• A min-max optimization is performed at each scale to jointly train generators and discriminators using the full loss function.

Experimental results

Research questions

• RQ1 Can adversarial patches be effectively generated using only a single image, without requiring large-scale pretraining data?
• RQ2 How can adversarial patches be made visually inconspicuous while maintaining high attack success rates?
• RQ3 To what extent can the proposed method achieve transferability across diverse victim models in black-box settings?
• RQ4 How does the proposed method compare to existing approaches in terms of human detectability?

Key findings

• The proposed method achieves a 99.58% average attack success rate in white-box settings across 10 ImageNet classes.
• In black-box settings, the method maintains strong transferability, achieving 99.25% success rate on GoogleNet and 90.46% on MobileNetV2.
• Human evaluation shows only 36.96% of images with generated patches were detected as synthetic, significantly lower than Google Patch (93.63%) and PS-GAN (89.90%).
• The method successfully generates visually inconspicuous patches that are perceptually consistent with the background, as confirmed by qualitative examples in Figure 3.
• The ablation study confirms the importance of multi-scale training and context-aware discrimination in improving patch realism and attack effectiveness.