[Paper Review] Quantifying Privacy Risks of Prompts in Visual Prompt Learning

Yixin Wu, Rui Wen+5 more|arXiv (Cornell University)|Oct 18, 2023

Adversarial Robustness in Machine LearningComputer Science3 citations

TL;DR

This paper conducts the first comprehensive privacy risk assessment of visual prompts in prompt learning, demonstrating that prompts are vulnerable to both property inference and membership inference attacks. Despite the reduced parameter count, prompts leak sensitive information about their training data, with successful attacks possible even under relaxed assumptions, highlighting critical privacy risks in prompt-as-a-service deployments.

ABSTRACT

Large-scale pre-trained models are increasingly adapted to downstream tasks through a new paradigm called prompt learning. In contrast to fine-tuning, prompt learning does not update the pre-trained model's parameters. Instead, it only learns an input perturbation, namely prompt, to be added to the downstream task data for predictions. Given the fast development of prompt learning, a well-generalized prompt inevitably becomes a valuable asset as significant effort and proprietary data are used to create it. This naturally raises the question of whether a prompt may leak the proprietary information of its training data. In this paper, we perform the first comprehensive privacy assessment of prompts learned by visual prompt learning through the lens of property inference and membership inference attacks. Our empirical evaluation shows that the prompts are vulnerable to both attacks. We also demonstrate that the adversary can mount a successful property inference attack with limited cost. Moreover, we show that membership inference attacks against prompts can be successful with relaxed adversarial assumptions. We further make some initial investigations on the defenses and observe that our method can mitigate the membership inference attacks with a decent utility-defense trade-off but fails to defend against property inference attacks. We hope our results can shed light on the privacy risks of the popular prompt learning paradigm. To facilitate the research in this direction, we will share our code and models with the community.

Motivation & Objective

• To investigate whether visual prompts in prompt learning leak proprietary information from their training data.
• To assess the feasibility of property inference and membership inference attacks on prompts as a new privacy threat surface.
• To evaluate existing attack methods adapted to the input-level privacy risk of prompts, distinct from model-level vulnerabilities.
• To explore initial defense strategies, particularly noise-based mitigation, for protecting prompts against inference attacks.

Proposed method

• Adapted established property inference attack methods—using a meta-classifier trained on shadow prompts—to infer sensitive attributes of the prompt's training dataset.
• Applied membership inference attacks (neural network-based, metric-based, and gradient-based) to determine whether a given image was part of the prompt's training data.
• Trained multiple shadow prompts using the same data distribution as the target prompt to build attack models.
• Evaluated defense mechanisms by adding Gaussian noise to prompts and measuring utility and attack success rates.
• Used standard vision benchmarks (e.g., CUB-200-2011, Cifar-10) to evaluate attack and defense performance under realistic settings.
• Shared code and models to enable reproducibility and further research in prompt privacy.

Figure 1 : Overview of prompt usage and inference attacks. The prompt is a pixel patch. The prompted image is an original image with an added prompt. Property inference infers sensitive properties of the target prompt’s training dataset that the PaaS provider does not intend to disclose. Membership

Experimental results

Research questions

• RQ1 To what extent can an adversary infer sensitive properties of the training data used to create a visual prompt?
• RQ2 Can membership inference attacks determine whether a specific image was part of the prompt's training dataset?
• RQ3 How effective are existing privacy attacks when applied to prompts, given their low parameter count and input-level nature?
• RQ4 Can noise-based defenses mitigate membership inference attacks while preserving prompt utility?
• RQ5 Why do current defenses fail to protect against property inference attacks in the prompt learning paradigm?

Key findings

• Visual prompts are vulnerable to property inference attacks, with adversaries able to successfully infer sensitive attributes of the training dataset using only a few shadow prompts.
• Membership inference attacks on prompts are effective even under relaxed adversarial assumptions, such as limited access to the training data distribution.
• Adding Gaussian noise to prompts mitigates membership inference attacks with a reasonable trade-off between defense effectiveness and model utility.
• DP-SGD fails to preserve prompt utility while providing insufficient protection against property inference attacks.
• The proposed defense strategy does not generalize to property inference attacks, indicating a critical gap in current defense approaches for input-level privacy risks.
• The results demonstrate that prompt learning, despite its efficiency, introduces new privacy risks at the input level that are not adequately addressed by existing defenses.

Figure 2 : Overview of visual prompt learning (VPL). We learn an input prompt via back-propagation [ 4 ] at the input transformation stage. We apply hard-coded mapping [ 13 ] to map the pre-trained model’s outputs into the target labels at the output transformation stage.