[论文解读] Measuring the Success of Diffusion Models at Imitating Human Artists

Stephen Casper, Zifan Guo+6 more|arXiv (Cornell University)|Jul 8, 2023

Generative Adversarial Networks and Image SynthesisComputer Science被引用 3

一句话总结

本文提出了一种基于CLIP的零样本方法，用于定量衡量扩散模型（如Stable Diffusion）模仿特定人类艺术家的能力。通过提示模型生成具有特定艺术家风格的图像，并使用CLIP对输出进行分类，作者发现Stable Diffusion在81.0%的情况下能正确识别出艺术家，表明其具备广泛模仿个体艺术家风格的能力。

ABSTRACT

Modern diffusion models have set the state-of-the-art in AI image generation. Their success is due, in part, to training on Internet-scale data which often includes copyrighted work. This prompts questions about the extent to which these models learn from, imitate, or copy the work of human artists. This work suggests that tying copyright liability to the capabilities of the model may be useful given the evolving ecosystem of generative models. Specifically, much of the legal analysis of copyright and generative systems focuses on the use of protected data for training. As a result, the connections between data, training, and the system are often obscured. In our approach, we consider simple image classification techniques to measure a model's ability to imitate specific artists. Specifically, we use Contrastive Language-Image Pretrained (CLIP) encoders to classify images in a zero-shot fashion. Our process first prompts a model to imitate a specific artist. Then, we test whether CLIP can be used to reclassify the artist (or the artist's work) from the imitation. If these tests match the imitation back to the original artist, this suggests the model can imitate that artist's expression. Our approach is simple and quantitative. Furthermore, it uses standard techniques and does not require additional training. We demonstrate our approach with an audit of Stable Diffusion's capacity to imitate 70 professional digital artists with copyrighted work online. When Stable Diffusion is prompted to imitate an artist from this set, we find that the artist can be identified from the imitation with an average accuracy of 81.0%. Finally, we also show that a sample of the artist's work can be matched to these imitation images with a high degree of statistical reliability. Overall, these results suggest that Stable Diffusion is broadly successful at imitating individual human artists.

研究动机与目标

• 开发一种实用且定量的方法，用于评估扩散模型模仿特定人类艺术家的能力。
• 在评估AI生成艺术的版权责任时，将关注点从训练数据转向模型能力本身。
• 证明标准机器学习技术可用于分析与AI模仿相关的法律主张。
• 评估Stable Diffusion在多大程度上能够再现个别数字艺术家的视觉风格。
• 提供一种可复现的基准，使用现成模型和标准技术来衡量AI模仿的成功程度。

提出的方法

• 使用Stable Diffusion v1.5，针对每位艺术家生成10张图像，提示词格式为“Artwork from <artist name>”
• 使用CLIP的图像编码器和文本编码器对生成的图像和文本标签（包括艺术家姓名和默认标签）进行编码
• 通过计算图像嵌入与文本嵌入之间的余弦相似度，选择相似度最高的标签作为分类结果
• 对每位艺术家重复分类实验10次，以减少方差并计算平均准确率
• 将结果与两个基线进行比较：随机艺术家姓名（准确率为8.6%）和随机猜测（准确率为1.4%）
• 进行第二项实验，利用CLIP嵌入和统计显著性检验（使用Bonferroni校正的秩和检验），将真实艺术家作品与生成的模仿作品进行匹配

Figure 1: Identifying human artists from Stable Diffusion Imitations. For each artist, we generate an imitation image from Stable Diffusion with the prompt “Artwork from $<$ artist name $>$ .” Next, we encode the image with a CLIP image encoder (Radford et al., 2021 ) . We also encode labels corresp

实验结果

研究问题

• RQ1 扩散模型（如Stable Diffusion）在多大程度上能够模仿个别人类艺术家的视觉风格？
• RQ2 像CLIP这样的标准零样本图像分类技术能否检测并识别出AI生成图像所对应的特定艺术家？
• RQ3 该模型的模仿能力与随机机会或基线模型相比如何？
• RQ4 AI生成的模仿作品与真实艺术家作品之间的相似性是否具有统计显著性？
• RQ5 是否可以不依赖训练数据，而仅通过模型输出行为来独立衡量模仿的成功程度？

主要发现

• Stable Diffusion在70位专业数字艺术家中，对生成图像的艺术家分类平均准确率达到81.0%。
• 在70位艺术家中，有69位在多数试验中被正确识别，表明其具备广泛且一致的模仿能力。
• 随机姓名基线仅达到8.6%的准确率，随机猜测仅为1.4%，证实结果并非偶然所致。
• 在将真实作品与生成模仿作品匹配时，90%的艺术家（63/70）显示出统计显著的相似性（p < 0.05，经Bonferroni校正）。
• 当在LAION数据集中代表性最高的250位艺术家上测试时，也观察到相似的81.2%准确率，表明结果在不同艺术家集合中具有鲁棒性。
• 该方法成功识别出模仿作品与目标艺术家的真实作品更为相似，而非与其他艺术家的作品相似，证实了模仿的保真度。

Figure 2: Example images generated by Stable Diffusion from prompts of the form “Artwork from $<$ artist’s name $>$ ”. Using the method depicted in Figure 1 , we show that the artists used in the prompts can often be classified from these imitations of their work.