[Paper Review] Measuring the Success of Diffusion Models at Imitating Human Artists

Stephen Casper, Zifan Guo+6 more|arXiv (Cornell University)|Jul 8, 2023

Generative Adversarial Networks and Image SynthesisComputer Science3 citations

TL;DR

This paper proposes a zero-shot, CLIP-based method to quantitatively measure how well diffusion models like Stable Diffusion imitate specific human artists. By prompting the model to generate images in an artist's style and using CLIP to classify the output, the authors show that Stable Diffusion correctly identifies the artist in 81.0% of cases, demonstrating broad capability to imitate individual artists' styles.

ABSTRACT

Modern diffusion models have set the state-of-the-art in AI image generation. Their success is due, in part, to training on Internet-scale data which often includes copyrighted work. This prompts questions about the extent to which these models learn from, imitate, or copy the work of human artists. This work suggests that tying copyright liability to the capabilities of the model may be useful given the evolving ecosystem of generative models. Specifically, much of the legal analysis of copyright and generative systems focuses on the use of protected data for training. As a result, the connections between data, training, and the system are often obscured. In our approach, we consider simple image classification techniques to measure a model's ability to imitate specific artists. Specifically, we use Contrastive Language-Image Pretrained (CLIP) encoders to classify images in a zero-shot fashion. Our process first prompts a model to imitate a specific artist. Then, we test whether CLIP can be used to reclassify the artist (or the artist's work) from the imitation. If these tests match the imitation back to the original artist, this suggests the model can imitate that artist's expression. Our approach is simple and quantitative. Furthermore, it uses standard techniques and does not require additional training. We demonstrate our approach with an audit of Stable Diffusion's capacity to imitate 70 professional digital artists with copyrighted work online. When Stable Diffusion is prompted to imitate an artist from this set, we find that the artist can be identified from the imitation with an average accuracy of 81.0%. Finally, we also show that a sample of the artist's work can be matched to these imitation images with a high degree of statistical reliability. Overall, these results suggest that Stable Diffusion is broadly successful at imitating individual human artists.

Motivation & Objective

• To develop a practical, quantitative method to assess how well diffusion models imitate specific human artists.
• To shift focus from training data to model capabilities when evaluating copyright liability for AI-generated art.
• To demonstrate that standard machine learning techniques can be used to analyze legal claims around AI imitation.
• To evaluate the extent to which Stable Diffusion can reproduce the visual style of individual digital artists.
• To provide a reproducible benchmark for measuring AI imitation success using off-the-shelf models and standard techniques.

Proposed method

• Use Stable Diffusion v1.5 to generate 10 images per artist using prompts of the form 'Artwork from <artist name>'
• Encode generated images and text labels (including artist names and default labels) using CLIP's image and text encoders
• Classify each generated image by computing cosine similarity between image and text embeddings, selecting the label with highest similarity
• Repeat the classification experiment 10 times per artist to reduce variance and compute average accuracy
• Compare results against two baselines: random artist names (8.6% accuracy) and random guessing (1.4% accuracy)
• Conduct a second experiment matching real artist artworks to generated imitations using CLIP embeddings and statistical significance testing (rank sum test with Bonferroni correction)

Figure 1: Identifying human artists from Stable Diffusion Imitations. For each artist, we generate an imitation image from Stable Diffusion with the prompt “Artwork from $<$ artist name $>$ .” Next, we encode the image with a CLIP image encoder (Radford et al., 2021 ) . We also encode labels corresp

Experimental results

Research questions

• RQ1 To what extent can diffusion models like Stable Diffusion imitate the visual style of individual human artists?
• RQ2 Can standard zero-shot image classification techniques like CLIP detect and attribute AI-generated images to specific artists?
• RQ3 How does the model’s imitation capability compare to random chance or baseline models?
• RQ4 Are the similarities between AI-generated imitations and real artist works statistically significant?
• RQ5 Can the success of imitation be measured independently of training data, focusing instead on model output behavior?

Key findings

• Stable Diffusion correctly classifies the artist of generated images with an average accuracy of 81.0% across 70 professional digital artists.
• The model correctly identifies the artist in a plurality of trials for 69 out of 70 artists, indicating broad and consistent imitation capability.
• The random-name baseline achieved only 8.6% accuracy, and random guessing achieved 1.4%, confirming that the results are not due to chance.
• When matching real artworks to generated imitations, 90% of artists (63 out of 70) showed statistically significant similarity (p < 0.05 after Bonferroni correction).
• The results are robust across different artist sets, as a similar 81.2% accuracy was observed when testing on the 250 most represented artists in the LAION dataset.
• The method successfully identifies that imitations are more similar to the target artist’s real works than to those of other artists, confirming the fidelity of imitation.

Figure 2: Example images generated by Stable Diffusion from prompts of the form “Artwork from $<$ artist’s name $>$ ”. Using the method depicted in Figure 1 , we show that the artists used in the prompts can often be classified from these imitations of their work.