[Paper Review] The Stable Signature: Rooting Watermarks in Latent Diffusion Models

Pierre Fernandez, Guillaume Couairon+3 more|arXiv (Cornell University)|Mar 27, 2023

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes Stable Signature, a method to embed invisible watermarks directly into the latent diffusion model (LDM) by fine-tuning only the latent decoder with a binary signature. The approach enables robust, efficient, and secure detection of AI-generated images—even after cropping, resizing, or editing—achieving over 90% detection accuracy with a false positive rate below 10⁻⁶.

ABSTRACT

Generative image modeling enables a wide range of applications but raises ethical concerns about responsible deployment. This paper introduces an active strategy combining image watermarking and Latent Diffusion Models. The goal is for all generated images to conceal an invisible watermark allowing for future detection and/or identification. The method quickly fine-tunes the latent decoder of the image generator, conditioned on a binary signature. A pre-trained watermark extractor recovers the hidden signature from any generated image and a statistical test then determines whether it comes from the generative model. We evaluate the invisibility and robustness of the watermarks on a variety of generation tasks, showing that Stable Signature works even after the images are modified. For instance, it detects the origin of an image generated from a text prompt, then cropped to keep $10\%$ of the content, with $90$+$\%$ accuracy at a false positive rate below 10$^{-6}$.

Motivation & Objective

• To address ethical concerns around synthetic image authenticity by enabling reliable detection of AI-generated images.
• To overcome the limitations of post-hoc watermarking, which can be easily removed if the model is open-sourced.
• To integrate watermarking natively into the generation process without architectural changes or performance degradation.
• To ensure watermark robustness under real-world image manipulations such as cropping, resizing, and editing.
• To provide a secure, efficient, and compatible solution for model providers and content platforms to trace and monitor AI-generated content.

Proposed method

• Fine-tune only the latent decoder of a pre-trained LDM using a combination of perceptual loss and a hidden message loss from a watermark extractor.
• Use a pre-trained watermark extractor based on a simplified HiDDeN method to recover the embedded binary signature from generated images.
• Apply whitening to the watermark extractor's output to make bit distributions closer to i.i.d. Bernoulli, improving statistical reliability.
• Train the watermark extractor on a subset of real images to ensure the extracted message is statistically indistinguishable from random when not from a watermarked image.
• Use a statistical test with a threshold to determine whether a generated image contains the watermark, based on Hamming distance between extracted and expected messages.
• Perform end-to-end training with back-propagation to embed the signature during generation, preserving image quality and FID scores.

Figure 1: Overview. The latent decoder can be fine-tuned to preemptively embed a signature into all generated images.

Experimental results

Research questions

• RQ1 Can watermarking be seamlessly integrated into the latent diffusion model generation process without modifying the architecture or diffusion process?
• RQ2 How robust is the watermark to common image manipulations such as cropping, resizing, and inpainting?
• RQ3 Can the watermark be detected with high accuracy while maintaining a very low false positive rate?
• RQ4 Does the watermarking process degrade the perceptual quality or FID score of generated images?
• RQ5 How does the proposed method compare in security and efficiency to post-hoc watermarking techniques?

Key findings

• The method achieves over 90% detection accuracy on images cropped to just 10% of their original size, with a false positive rate below 10⁻⁶.
• The watermark remains robust even after complex edits such as inpainting and super-resolution, with bit accuracy significantly above random chance in challenging scenarios.
• The FID score of generated images remains unchanged after watermarking, confirming no degradation in image quality.
• The watermark extractor achieves near-perfect match between theoretical and empirical false positive rates, validating the statistical model's accuracy.
• The approach is computationally efficient and compatible with a wide range of LDM-based applications, including text-to-image, inpainting, and image editing.
• The method outperforms post-hoc watermarking in security, as the watermark cannot be removed without retraining the model, even if the model is open-sourced.

Figure 2: Steps of the method. (a) We pre-train a watermark encoder $\mathcal{W}_{E}$ and extractor $\mathcal{W}$ , to extract binary messages. (b) We fine-tune the decoder $\mathcal{D}$ of the LDM’s auto-encoder with a fixed signature $m$ such that all the generated images (c) lead to $m$ through $