[Paper Review] DUAW: Data-free Universal Adversarial Watermark against Stable Diffusion Customization

Xiaoyu Ye, Hao Huang+2 more|arXiv (Cornell University)|Aug 19, 2023

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes DUAW, a data-free universal adversarial watermark that disrupts Stable Diffusion model customization by perturbing the variational autoencoder (VAE) during fine-tuning, thereby inducing severe visual distortions in generated images. Trained on synthetic images from a large language model and a pretrained SD model, DUAW invisibly protects copyrighted images across multiple SD versions and customization methods, achieving 96.43% accuracy in identifying infringing outputs with a simple classifier.

ABSTRACT

Stable Diffusion (SD) customization approaches enable users to personalize SD model outputs, greatly enhancing the flexibility and diversity of AI art. However, they also allow individuals to plagiarize specific styles or subjects from copyrighted images, which raises significant concerns about potential copyright infringement. To address this issue, we propose an invisible data-free universal adversarial watermark (DUAW), aiming to protect a myriad of copyrighted images from different customization approaches across various versions of SD models. First, DUAW is designed to disrupt the variational autoencoder during SD customization. Second, DUAW operates in a data-free context, where it is trained on synthetic images produced by a Large Language Model (LLM) and a pretrained SD model. This approach circumvents the necessity of directly handling copyrighted images, thereby preserving their confidentiality. Once crafted, DUAW can be imperceptibly integrated into massive copyrighted images, serving as a protective measure by inducing significant distortions in the images generated by customized SD models. Experimental results demonstrate that DUAW can effectively distort the outputs of fine-tuned SD models, rendering them discernible to both human observers and a simple classifier.

Motivation & Objective

• To address the growing risk of copyright infringement via Stable Diffusion customization, where users can plagiarize subjects or styles from copyrighted images.
• To develop a universal, invisible watermark that protects diverse copyrighted images without requiring access to the original images.
• To ensure robust protection across multiple Stable Diffusion versions and customization techniques (e.g., DreamBooth, LoRA) without altering model weights.
• To maintain watermark invisibility while maximizing distortion in fine-tuned model outputs, even under image preprocessing attacks.
• To enable data-free training by generating synthetic training data via LLM and pretrained SD models, preserving confidentiality of protected images.

Proposed method

• DUAW is trained using an optimizer-based adversarial attack that minimizes the similarity between original images and VAE-reconstructed images when the watermark is applied.
• The watermark is optimized to disrupt the VAE's encoding and decoding process, which remains frozen during SD fine-tuning, ensuring consistent distortion in generated outputs.
• Synthetic training images are generated using a large language model (e.g., ChatGPT) and a pretrained Stable Diffusion v2.1 model, enabling data-free training without accessing real copyrighted images.
• The method targets the VAE component specifically, leveraging the observation that minor perturbations in training images are preserved and amplified in fine-tuned model outputs.
• The watermark is applied universally across diverse images and tested for transferability across different VAE variants (base, ft-ema, ft-mse) and SD model versions.
• Robustness is evaluated under common image corruptions (Gaussian blur, JPEG compression, random noise), with protection success rate reported as a key metric.

Experimental results

Research questions

• RQ1 Can a universal adversarial watermark be crafted to distort outputs of fine-tuned Stable Diffusion models without access to real copyrighted images?
• RQ2 Does disrupting the VAE during SD customization effectively degrade the quality of generated images across different customization methods (e.g., DreamBooth, LoRA)?
• RQ3 Can a data-free training approach using LLM-generated synthetic images achieve effective watermarking while preserving image confidentiality?
• RQ4 How robust is the watermark against common image preprocessing attacks such as JPEG compression and Gaussian blur?
• RQ5 To what extent does the watermark generalize across different VAE variants and future versions of Stable Diffusion models?

Key findings

• DUAW achieves a 96.43% success rate in identifying infringing images using a simple ViT classifier, demonstrating high effectiveness in detecting model-generated distortions.
• The watermark induces significant visual degradation in generated images, with human observers clearly perceiving distortions across diverse prompts and customization methods.
• DUAW shows strong transferability across different VAE variants: when trained on ft-mse, it achieves 99.05% protection success rate on base VAE and 99.90% on ft-ema, indicating broad compatibility.
• The watermark remains robust under common image corruptions—Gaussian blur and JPEG compression slightly reduce distortion but do not compromise overall protection efficacy.
• At an epsilon value of 0.05, DUAW maintains high invisibility (SSIM ≈ 0.98) while still causing effective distortion, balancing imperceptibility and protection.
• The method is effective across multiple SD model versions, including SD v2.1 and SD XL 1.0, with consistent performance, suggesting potential for future model protection.