[Paper Review] Piracy Resistant Watermarks for Deep Neural Networks

Huiying Li, Emily Wenger+3 more|arXiv (Cornell University)|Oct 2, 2019

Adversarial Robustness in Machine Learning43 references31 citations

TL;DR

This paper proposes null embedding, a novel method to embed piracy-resistant watermarks into deep neural networks by enforcing a strong dependency between normal classification accuracy and the watermark during initial training. Unlike prior methods reliant on incremental training, null embedding prevents attackers from removing or adding new watermarks without destroying model performance, achieving robust piracy resistance across diverse models and tasks.

ABSTRACT

As companies continue to invest heavily in larger, more accurate and more robust deep learning models, they are exploring approaches to monetize their models while protecting their intellectual property. Model licensing is promising, but requires a robust tool for owners to claim ownership of models, i.e. a watermark. Unfortunately, current designs have not been able to address piracy attacks, where third parties falsely claim model ownership by embedding their own "pirate watermarks" into an already-watermarked model. We observe that resistance to piracy attacks is fundamentally at odds with the current use of incremental training to embed watermarks into models. In this work, we propose null embedding, a new way to build piracy-resistant watermarks into DNNs that can only take place at a model's initial training. A null embedding takes a bit string (watermark value) as input, and builds strong dependencies between the model's normal classification accuracy and the watermark. As a result, attackers cannot remove an embedded watermark via tuning or incremental training, and cannot add new pirate watermarks to already watermarked models. We empirically show that our proposed watermarks achieve piracy resistance and other watermark properties, over a wide range of tasks and models. Finally, we explore a number of adaptive counter-measures, and show our watermark remains robust against a variety of model modifications, including model fine-tuning, compression, and existing methods to detect/remove backdoors. Our watermarked models are also amenable to transfer learning without losing their watermark properties.

Motivation & Objective

• Address the critical lack of piracy resistance in existing DNN watermarking schemes, where attackers can embed counterfeit watermarks via incremental training.
• Overcome the fundamental vulnerability of current watermarking methods, which rely on incremental training and thus allow attackers to overwrite or add watermarks.
• Design a watermarking system that is unforgeable, persistent, and verifiable, even under model fine-tuning, compression, or transfer learning.
• Ensure that attempts to embed a new watermark into a watermarked model result in catastrophic loss of classification accuracy, deterring malicious use.

Proposed method

• Introduce null embedding: a technique that embeds a watermark bit string as a constraint during initial model training, creating a strong dependency between the watermark and the model’s normal classification behavior.
• Use public key cryptography and verifiable signatures to bind the watermark bit string securely to the model owner, enabling authentication and verification.
• Embed the watermark by modifying the optimization process during training so that the model’s weights are constrained to satisfy both accurate classification and watermark consistency.
• Prevent incremental training from altering or adding watermarks by making any such attempt conflict with the original watermark constraint, thereby degrading model accuracy.
• Leverage the irreversibility of the initial training phase to ensure that watermarks cannot be removed or replaced without retraining from scratch.
• Design the watermark to remain intact through transfer learning, model compression, and other common model modifications, preserving ownership proof.

Experimental results

Research questions

• RQ1 Can a DNN watermarking scheme be designed to be truly resistant to piracy attacks, where an attacker embeds a counterfeit watermark into an already-watermarked model?
• RQ2 Why are existing watermarking techniques vulnerable to piracy attacks, and what architectural or training property enables this vulnerability?
• RQ3 Can a watermark be embedded in a way that prevents removal or replacement via incremental training or fine-tuning?
• RQ4 How robust is the proposed watermark under various model modifications, including compression, transfer learning, and backdoor removal techniques?
• RQ5 Can the watermark be preserved and verified even when the model is extracted or retrained using out-of-distribution data?

Key findings

• Existing watermarking methods, including regularizer-based and artifact-based approaches, are vulnerable to piracy attacks, as attackers can successfully embed new watermarks via incremental training.
• Null embedding successfully prevents any incremental training from modifying or adding watermarks, as such attempts degrade the model’s normal classification accuracy below acceptable thresholds.
• Piracy attacks against null-embedded models result in performance loss comparable to training from scratch, making such attacks computationally and practically infeasible.
• The watermark remains robust under model fine-tuning, compression, and existing backdoor detection/removal techniques, preserving ownership proof.
• Watermarked models remain compatible with transfer learning, maintaining watermark integrity across different downstream tasks.
• Model extraction attacks using out-of-distribution data (e.g., ImageNet, YouTube Faces) require significantly more data and computational cost than training from scratch, with accuracy gains only achievable at 255% of the original dataset size.