[Paper Review] Exploring Structure Consistency for Deep Model Watermarking

Jie Zhang, Dongdong Chen+6 more|arXiv (Cornell University)|Aug 5, 2021

Adversarial Robustness in Machine Learning32 references4 citations

TL;DR

This paper proposes a novel deep model watermarking method based on 'structure consistency,' which embeds watermarks into physically consistent image structures like edges or semantic regions (e.g., eyes) to enhance robustness against data augmentation attacks. Unlike prior methods relying on whole-image consistency, this approach maintains watermark integrity under rotations, crops, and resizes, achieving superior robustness in image processing tasks such as deraining and portrait generation with over 90% extraction accuracy under aggressive augmentations.

ABSTRACT

The intellectual property (IP) of Deep neural networks (DNNs) can be easily ``stolen'' by surrogate model attack. There has been significant progress in solutions to protect the IP of DNN models in classification tasks. However, little attention has been devoted to the protection of DNNs in image processing tasks. By utilizing consistent invisible spatial watermarks, one recent work first considered model watermarking for deep image processing networks and demonstrated its efficacy in many downstream tasks. Nevertheless, it highly depends on the hypothesis that the embedded watermarks in the network outputs are consistent. When the attacker uses some common data augmentation attacks (e.g., rotate, crop, and resize) during surrogate model training, it will totally fail because the underlying watermark consistency is destroyed. To mitigate this issue, we propose a new watermarking methodology, namely ``structure consistency'', based on which a new deep structure-aligned model watermarking algorithm is designed. Specifically, the embedded watermarks are designed to be aligned with physically consistent image structures, such as edges or semantic regions. Experiments demonstrate that our method is much more robust than the baseline method in resisting data augmentation attacks for model IP protection. Besides that, we further test the generalization ability and robustness of our method to a broader range of circumvention attacks.

Motivation & Objective

• To address the fragility of existing watermarking schemes in deep image processing models under data augmentation attacks.
• To identify the root cause of failure in whole-image consistency-based watermarking when subjected to common augmentations like cropping and rotation.
• To develop a new watermarking methodology that preserves watermark consistency through physically meaningful image structures such as edges or semantic regions.
• To design a structure-aligned watermarking framework that maintains high visual quality and robustness across diverse image processing tasks.

Proposed method

• Proposes a new watermarking methodology called 'structure consistency,' where watermarks are encoded into physically stable image structures like edges or semantic regions (e.g., eyes).
• Designs a four-module framework: watermark bit encoder, embedding network, extracting network, and decoder, all trained end-to-end to preserve watermark integrity.
• Uses constant color values embedded into detected structures (via Canny or semantic segmentation) to create structure-aligned watermarks that remain consistent under geometric transformations.
• Employs an incremental training strategy that gradually introduces augmentation operators and loss constraints to improve robustness during training.
• Leverages the extracting network to detect and recover watermarks only from watermarked images, ensuring no false positives on clean inputs.
• Applies the framework to diverse tasks including deraining, artistic portrait generation, and X-ray debone, demonstrating generalization across different physical structures and applications.

Experimental results

Research questions

• RQ1 Why does the whole-image consistency-based watermarking method fail under common data augmentation attacks such as random cropping and rotation?
• RQ2 Can watermark consistency be preserved if watermarks are aligned with physically stable image structures like edges or semantic regions instead of fixed image positions?
• RQ3 How does structure consistency improve robustness against surrogate model attacks that use augmented training data?
• RQ4 To what extent can the proposed method generalize across different image processing tasks and physical structure types (e.g., global edges vs. local semantic regions)?
• RQ5 How resilient is the structure-consistent watermarking scheme to advanced circumvention techniques such as Neural Cleanse, fine-tuning, and domain-adversarial training?

Key findings

• The proposed structure-consistent watermarking method achieves over 90% watermark extraction accuracy under aggressive data augmentation, significantly outperforming the baseline whole-image consistency method, which fails completely under such attacks.
• The method maintains high robustness even when 50% of the surrogate model training data is self-labeled clean data, with a 78% success rate in watermark extraction despite the destruction of consistency constraints.
• Neural Cleanse fails to detect the watermark because the trigger is not input-agnostic or static in location and pattern, as assumed by the attack, due to the global and structure-aligned nature of the watermark.
• Fine-tuning with a small amount of clean data reduces the watermark extraction success rate to 78%, but the surrogate model’s performance degrades significantly (PSNR drops from 32.3 to 28.9), reducing the attack’s practicality.
• The framework generalizes well across diverse tasks: it successfully extracts watermarks in both deraining and artistic portrait generation, with visual results showing minimal perceptual distortion.
• Even after watermark overwriting, the original watermark can still be extracted, and the surrogate model’s performance is substantially degraded, indicating strong resilience to post-attack manipulation.