Skip to main content
QUICK REVIEW

[论文解读] Nightshade: Prompt-Specific Poisoning Attacks on Text-to-Image Generative Models

Shawn Shan, Wenxin Ding|arXiv (Cornell University)|Oct 20, 2023
Advanced Data Storage TechnologiesComputer Science被引用 12
一句话总结

本文表明文本到图像扩散模型容易受到针对提示的特定数据中毒,提出 Nightshade——一种隐蔽、优化的中毒方法,在样本相对较少的情况下也有效,并且能够对相关提示产生穿透效应。

ABSTRACT

Data poisoning attacks manipulate training data to introduce unexpected behaviors into machine learning models at training time. For text-to-image generative models with massive training datasets, current understanding of poisoning attacks suggests that a successful attack would require injecting millions of poison samples into their training pipeline. In this paper, we show that poisoning attacks can be successful on generative models. We observe that training data per concept can be quite limited in these models, making them vulnerable to prompt-specific poisoning attacks, which target a model's ability to respond to individual prompts. We introduce Nightshade, an optimized prompt-specific poisoning attack where poison samples look visually identical to benign images with matching text prompts. Nightshade poison samples are also optimized for potency and can corrupt an Stable Diffusion SDXL prompt in <100 poison samples. Nightshade poison effects "bleed through" to related concepts, and multiple attacks can composed together in a single prompt. Surprisingly, we show that a moderate number of Nightshade attacks can destabilize general features in a text-to-image generative model, effectively disabling its ability to generate meaningful images. Finally, we propose the use of Nightshade and similar tools as a last defense for content creators against web scrapers that ignore opt-out/do-not-crawl directives, and discuss possible implications for model trainers and content creators.

研究动机与目标

  • 证明基于扩散的文本到图像模型容易受到针对特定提示的中毒攻击。
  • 表明低训练数据密度(概念稀疏性)使得针对提示的中毒有效。
  • 开发 Nightshade,一种经过优化的中毒方法,样本很少时仍然隐蔽且高效。
  • 考察对相关概念的渗透效应及在模型之间的迁移性。
  • 讨论其作为内容创作者版权保护工具的潜在正当用途。

提出的方法

  • 分析大规模扩散数据集中的概念稀疏性,并量化概念的词语及语义频率。
  • 评估简单的脏标签中毒攻击,以较少的中毒数据(例如 500–1000 个样本)建立基线有效性。
  • 通过将清晰传达目标概念的文本提示与一个不相关的目的地概念的图像结合来开发 Nightshade,然后通过清洁标签中毒技术扰动图像,使其看起来自然。
  • 为目的地概念生成锚定图像,并应用有约束的扰动,使被污染的图像在特征空间中与目标概念对齐(LPIPS 预算)。
  • 使用梯度式直觉在最大化污染数据对模型训练中与目标概念相关的影响的同时,保持视觉对齐以避免检测。
Figure 1: Overview of prompt-specific poison attack. a) User generates poison data (text and image pairs) designed to corrupt a given concept $C$ , then posts it online; b) Model trainer scrapes data from online webpages to train its generative model; c) Given prompts of $C$ , poisoned model generat
Figure 1: Overview of prompt-specific poison attack. a) User generates poison data (text and image pairs) designed to corrupt a given concept $C$ , then posts it online; b) Model trainer scrapes data from online webpages to train its generative model; c) Given prompts of $C$ , poisoned model generat

实验结果

研究问题

  • RQ1在没有访问训练管道的情况下,针对提示的中毒攻击能否在基于扩散的文本到图像模型上成功?
  • RQ2训练数据中的概念稀疏性如何影响中毒攻击的可行性和强度?
  • RQ3Nightshade 方法在在保持隐蔽的同时减少所需的中毒样本数量方面有多有效?
  • RQ4Nightshade 攻击是否对语义相关的提示存在渗透效应,并在多个目标之间产生累积效应?
  • RQ5Nightshade 攻击在不同模型之间的可迁移性以及对防御的抵抗力如何?

主要发现

  • 针对提示的中毒攻击可以在相对较少的中毒样本下取得高成功率,利用概念稀疏性。
  • 在某些设置中,简单的脏标签中毒攻击可以用 500–1000 个样本破坏目标概念。
  • Nightshade 进一步提高了效力,在某些情况下仅用约 100 个样本就能实现有效中毒。
  • Nightshade 的中毒效应会渗透到相关概念,当一个提示中同时污染多个概念时会叠加。
  • 被污染的攻击在模型之间具有可迁移性并抵御若干防御;广泛的中毒(如跨越许多提示)可能破坏模型的通用特征。
Figure 2: Demonstrating concept sparsity in terms of word and semantic frequencies in LAION-Aesthetic. Both show a long-tail distribution. Note the log scale on both Y axes.
Figure 2: Demonstrating concept sparsity in terms of word and semantic frequencies in LAION-Aesthetic. Both show a long-tail distribution. Note the log scale on both Y axes.

阅读本论文

在 Nubint 阅读器中打开原文,边读边就任意段落提问。

在 Nubint 中阅读本论文

本解读由 AI 根据论文原文生成,引用前请核对原文。