[Paper Review] BlackMarks: Blackbox Multibit Watermarking for Deep Neural Networks

Huili Chen, Bita Darvish Rouhani+1 more|arXiv (Cornell University)|Mar 31, 2019

Advanced Steganography and Watermarking Techniques32 references41 citations

TL;DR

BlackMarks introduces the first end-to-end black-box framework for multi-bit watermarking of DNNs, embedding a binary signature into model outputs via fine-tuning and extracting it through query keys with low overhead and high robustness.

ABSTRACT

Deep Neural Networks have created a paradigm shift in our ability to comprehend raw data in various important fields ranging from computer vision and natural language processing to intelligence warfare and healthcare. While DNNs are increasingly deployed either in a white-box setting where the model internal is publicly known, or a black-box setting where only the model outputs are known, a practical concern is protecting the models against Intellectual Property (IP) infringement. We propose BlackMarks, the first end-to-end multi-bit watermarking framework that is applicable in the black-box scenario. BlackMarks takes the pre-trained unmarked model and the owner's binary signature as inputs and outputs the corresponding marked model with a set of watermark keys. To do so, BlackMarks first designs a model-dependent encoding scheme that maps all possible classes in the task to bit '0' and bit '1' by clustering the output activations into two groups. Given the owner's watermark signature (a binary string), a set of key image and label pairs are designed using targeted adversarial attacks. The watermark (WM) is then embedded in the prediction behavior of the target DNN by fine-tuning the model with generated WM key set. To extract the WM, the remote model is queried by the WM key images and the owner's signature is decoded from the corresponding predictions according to the designed encoding scheme. We perform a comprehensive evaluation of BlackMarks's performance on MNIST, CIFAR10, ImageNet datasets and corroborate its effectiveness and robustness. BlackMarks preserves the functionality of the original DNN and incurs negligible WM embedding runtime overhead as low as 2.054%.

Motivation & Objective

• Motivate IP protection for DNNs in black-box settings (MLaaS).
• Develop a scalable, multi-bit watermarking framework that operates without model internals.
• Design a model-dependent encoding of class outputs into bits and generate watermark keys via targeted adversarial attacks.
• Fine-tune the watermarked model to embed the watermark while preserving accuracy.
• Provide a robust extraction and verification procedure with low false positives/negatives.

Proposed method

• Encode owner signature into DNN output activations by clustering class outputs into two bit-clusters via K-means on class means (pre-softmax).
• Generate watermark key images and labels using targeted adversarial attacks aligned with the encoding scheme.
• Fine-tune the pre-trained model with a regularized loss combining standard cross-entropy and a WM-specific loss to embed the signature.
• Decode the owner signature from model predictions by querying with WM keys and applying the encoding scheme, yielding Bit Error Rate (BER).
• Guard against transferability of WM keys by selecting a larger initial key set and intersecting across unmarked model variants during key selection.
• Provide an efficiency analysis: one-time embedding overhead (as low as 2.054%) and black-box extraction costs.

Experimental results

Research questions

• RQ1 Can a black-box DNN watermark bolster IP protection with multi-bit capacity?
• RQ2 How can a model-dependent encoding map outputs to bits to support multi-bit watermarking without harming accuracy?
• RQ3 What is the robustness of such a watermark to fine-tuning, pruning, and overwriting in a black-box setting?
• RQ4 How can watermark integrity and credibility be quantified and ensured in black-box scenarios?

Key findings

• BlackMarks achieves high watermark detection with zero BER after embedding across MNIST, CIFAR-10, and ImageNet when using the specified key sets.
• The framework tolerates up to 95%, 80%, and 90% parameter pruning for MNIST, CIFAR-10, and ImageNet respectively without degrading detection (though excessive pruning harms accuracy).
• Watermarks remain detectable after model fine-tuning (up to 100 epochs in experiments) with BER zero for all benchmarks.
• Watermark overwriting with a new watermark does not prevent recovering the original watermark (BER remains zero).
• Embedding incurs low runtime overhead (as low as 2.054%), and the watermarking approach also appears to improve adversarial robustness; the authors note a side benefit of increased accuracy under several attacks.