[Paper Review] Protect, Show, Attend and Tell: Empowering Image Captioning Models with Ownership Protection

Jian Han Lim, Chee Seng Chan+3 more|arXiv (Cornell University)|Aug 25, 2020

Multimodal Machine Learning Applications29 references4 citations

TL;DR

This paper proposes a novel key-based ownership protection mechanism for image captioning models by embedding a secret key into the hidden memory state of recurrent neural networks (RNNs), ensuring that only models with the valid key produce high-quality captions. The method preserves original model performance while rendering forged keys ineffective, achieving robust protection against removal and ambiguity attacks on MS-COCO and Flickr30k datasets.

ABSTRACT

By and large, existing Intellectual Property (IP) protection on deep neural networks typically i) focus on image classification task only, and ii) follow a standard digital watermarking framework that was conventionally used to protect the ownership of multimedia and video content. This paper demonstrates that the current digital watermarking framework is insufficient to protect image captioning tasks that are often regarded as one of the frontiers AI problems. As a remedy, this paper studies and proposes two different embedding schemes in the hidden memory state of a recurrent neural network to protect the image captioning model. From empirical points, we prove that a forged key will yield an unusable image captioning model, defeating the purpose of infringement. To the best of our knowledge, this work is the first to propose ownership protection on image captioning task. Also, extensive experiments show that the proposed method does not compromise the original image captioning performance on all common captioning metrics on Flickr30k and MS-COCO datasets, and at the same time it is able to withstand both removal and ambiguity attacks. Code is available at https://github.com/jianhanlim/ipr-imagecaptioning

Motivation & Objective

• To address the lack of intellectual property (IP) protection for image captioning models, which are often overlooked in existing watermarking frameworks.
• To develop a proactive, cost-effective, and timely IP protection mechanism that prevents unauthorized use of image captioning models.
• To ensure that only models with the correct secret key generate high-quality, semantically accurate captions, thereby deterring infringement.
• To demonstrate that the proposed method maintains near-identical performance to the original model when using the valid key, while degrading significantly with forged keys.
• To establish the first known solution for IP protection in image captioning, overcoming limitations of traditional watermarking methods designed for image classification.

Proposed method

• Embed a unique secret key into the hidden memory state of an RNN-based image captioning model during training, using two distinct embedding schemes.
• Modify the RNN cell’s hidden state computation to incorporate the secret key via a learnable transformation, ensuring the key is deeply integrated into the model’s internal representation.
• Use a signature verification mechanism during inference to validate the presence of the correct key, preventing model usage without authorization.
• Design the embedding such that any forged key disrupts the hidden state dynamics, leading to semantically incoherent or low-quality captions.
• Train the model end-to-end with the key embedded in the hidden state, preserving the original model’s performance on standard captioning metrics.
• Evaluate robustness under removal and ambiguity attacks by testing model performance with both valid and forged keys across multiple benchmarks.

Experimental results

Research questions

• RQ1 Can a secret key embedded in the hidden memory state of an RNN-based image captioning model effectively prevent unauthorized usage?
• RQ2 Does the proposed key-based protection mechanism preserve the original model’s performance on standard image captioning metrics?
• RQ3 How effective is the method against removal and ambiguity attacks, where an attacker attempts to extract or forge the secret key?
• RQ4 Can the method be generalized across different image captioning architectures, such as Up-Down and SCST?
• RQ5 How does the performance degradation with a forged key compare to the original model and watermarking-based approaches?

Key findings

• The proposed method maintains near-identical performance to the original model when using the valid key, with CIDEr-D scores of 101.93 (Up-Down) and 101.87 (SCST) on MS-COCO.
• With a forged key, the CIDEr-D score drops by 17.27% (to 84.33) on the Up-Down model and by 11.14% (to 90.53) on the SCST model, demonstrating strong degradation.
• The SPICE score drops from 19.90 to 16.53 with a forged key on the Up-Down model, indicating significant loss in referring expression accuracy.
• The method outperforms existing watermarking frameworks—such as passport-based watermarking—by producing more accurate and complete captions even under attack.
• The model with a forged key produces semantically incoherent or incorrect captions, effectively rendering it unusable for practical applications.
• The approach is robust across both MS-COCO and Flickr30k datasets, confirming generalization across different image captioning benchmarks.