[Paper Review] Intriguing Properties of Diffusion Models: An Empirical Study of the Natural Attack Capability in Text-to-Image Generative Models

Takami Sato, Justin Yue+3 more|arXiv (Cornell University)|Aug 30, 2023

Generative Adversarial Networks and Image SynthesisComputer Science3 citations

TL;DR

This paper identifies a novel natural attack capability in text-to-image diffusion models, termed Natural Denoising Diffusion (NDD) attack, where prompts remove human-essential robust features (e.g., color, shape) yet models still generate images that fool DNN object detectors. The NDD attack achieves 88% detection rate by models while remaining stealthy to 93% of human subjects, revealing that non-robust features embedded by diffusion models enable effective, model-agnostic, and transferable adversarial attacks.

ABSTRACT

Denoising probabilistic diffusion models have shown breakthrough performance to generate more photo-realistic images or human-level illustrations than the prior models such as GANs. This high image-generation capability has stimulated the creation of many downstream applications in various areas. However, we find that this technology is actually a double-edged sword: We identify a new type of attack, called the Natural Denoising Diffusion (NDD) attack based on the finding that state-of-the-art deep neural network (DNN) models still hold their prediction even if we intentionally remove their robust features, which are essential to the human visual system (HVS), through text prompts. The NDD attack shows a significantly high capability to generate low-cost, model-agnostic, and transferable adversarial attacks by exploiting the natural attack capability in diffusion models. To systematically evaluate the risk of the NDD attack, we perform a large-scale empirical study with our newly created dataset, the Natural Denoising Diffusion Attack (NDDA) dataset. We evaluate the natural attack capability by answering 6 research questions. Through a user study, we find that it can achieve an 88% detection rate while being stealthy to 93% of human subjects; we also find that the non-robust features embedded by diffusion models contribute to the natural attack capability. To confirm the model-agnostic and transferable attack capability, we perform the NDD attack against the Tesla Model 3 and find that 73% of the physically printed attacks can be detected as stop signs. Our hope is that the study and dataset can help our community be aware of the risks in diffusion models and facilitate further research toward robust DNN models.

Motivation & Objective

• To investigate whether text-to-image diffusion models inherently possess a natural capability to generate adversarial attacks by removing robust visual features essential for human perception.
• To systematically evaluate the effectiveness and stealthiness of such attacks using a large-scale empirical study.
• To construct and release the NDDA dataset—a comprehensive collection of 40,870 images generated with and without robust features—for benchmarking natural attack capabilities.
• To assess the real-world feasibility of these attacks by testing against a commodity autonomous driving vehicle.
• To raise awareness in the research community about the security risks posed by diffusion models and to stimulate development of more robust DNNs.

Proposed method

• Constructed the NDDA dataset by generating images using three state-of-the-art diffusion models (e.g., Stable Diffusion, DALL-E 2) with prompts that explicitly remove robust visual features (color, shape, text, pattern) essential for human recognition.
• Used contrastive image-text models like CLIP to condition image generation on text prompts while systematically degrading perceptual robustness.
• Evaluated the natural attack capability by measuring detection rates of 5 state-of-the-art object detectors on images with and without robust features.
• Conducted a user study with 93% of participants failing to detect the adversarial nature of the generated images, confirming stealthiness.
• Performed physical-world testing by printing adversarial stop signs and evaluating detection rates on a commodity autonomous driving vehicle.
• Analyzed feature contributions by examining the role of non-robust features in maintaining model predictions despite human-invisible perturbations.

Experimental results

Research questions

• RQ1 To what extent do state-of-the-art diffusion models retain the ability to generate adversarial examples that fool DNN object detectors even when robust visual features are removed via text prompts?
• RQ2 How stealthy are these adversarial examples to human observers, and to what extent do they evade human detection while maintaining high model detection rates?
• RQ3 Can the NDD attack be transferred across different DNN models and applied in the physical world, such as on real-world autonomous vehicles?
• RQ4 What role do non-robust features—imperceptible to humans but generalizable to DNNs—play in enabling the natural attack capability of diffusion models?
• RQ5 How generalizable is the NDD attack across different object classes and diffusion models?

Key findings

• The NDD attack achieves an 88% detection rate by object detectors on images of stop signs that lack essential robust features such as red color and octagonal shape.
• The attack remains stealthy to 93% of human subjects, who failed to detect the adversarial nature of the images in a user study.
• Even when all four robust features (color, shape, text, pattern) were removed from the prompt, 32% of generated stop signs were still detected as stop signs by object detectors.
• Physical-world testing showed that 73% of printed adversarial stop signs were detected as stop signs by a commodity autonomous driving vehicle.
• The study confirms that non-robust features embedded by diffusion models are a key enabler of the natural attack capability, as they generalize across models and tasks.
• The NDDA dataset, containing 40,870 images across 15 object classes and 6 diffusion models, is released to support future research on robustness and adversarial robustness in diffusion models.