[Paper Review] Not Every Image is Worth a Thousand Words: Quantifying Originality in Stable Diffusion

Adi Haviv, Shahar Sarfaty+4 more|arXiv (Cornell University)|Aug 15, 2024

Image and Signal Denoising Methods4 citations

TL;DR

This paper proposes a method to quantify originality in text-to-image diffusion models by measuring the number of textual inversion tokens required to reconstruct an image, with fewer tokens indicating higher familiarity and lower originality. The approach correlates token count with conceptual novelty, demonstrating that original images require more tokens to reconstruct, offering a data-free, model-internal metric aligned with legal definitions of originality.

ABSTRACT

This work addresses the challenge of quantifying originality in text-to-image (T2I) generative diffusion models, with a focus on copyright originality. We begin by evaluating T2I models' ability to innovate and generalize through controlled experiments, revealing that stable diffusion models can effectively recreate unseen elements with sufficiently diverse training data. Then, our key insight is that concepts and combinations of image elements the model is familiar with, and saw more during training, are more concisly represented in the model's latent space. We hence propose a method that leverages textual inversion to measure the originality of an image based on the number of tokens required for its reconstruction by the model. Our approach is inspired by legal definitions of originality and aims to assess whether a model can produce original content without relying on specific prompts or having the training data of the model. We demonstrate our method using both a pre-trained stable diffusion model and a synthetic dataset, showing a correlation between the number of tokens and image originality. This work contributes to the understanding of originality in generative models and has implications for copyright infringement cases.

Motivation & Objective

• To address the challenge of quantifying originality in text-to-image diffusion models, especially in the context of copyright law.
• To develop a method that assesses originality without requiring access to training data or specific prompts.
• To align the evaluation of model-generated content with legal standards of originality, emphasizing creativity and authorship.
• To demonstrate that model familiarity—measured via latent space tokenization—can serve as a proxy for originality.
• To provide a framework for auditing generative AI outputs for originality and genericity in real-world and synthetic applications.

Proposed method

• Leverage textual inversion to encode image content into a compact set of learnable tokens within the model’s latent space.
• Measure the number of tokens required to reconstruct a given image using textual inversion, with higher token counts indicating greater originality.
• Use a pre-trained Stable Diffusion model and a custom-trained model on synthetic data to validate the method across different settings.
• Correlate token count with image originality using both synthetic datasets (shapes, colors, sizes) and real-world images.
• Apply the DreamSim algorithm to compare reconstructed images with originals, ensuring fidelity and validity of the tokenization process.
• Base the framework on the hypothesis that frequently seen concepts in training require fewer tokens for representation, reflecting lower originality.

Figure 1 : Illustration of our approach for measuring image originality using multi-token textual inversion. Original images require more tokens for accurate reconstruction, while common images like Van Gogh’s ”Starry Night” need only one token.

Experimental results

Research questions

• RQ1 Can text-to-image diffusion models be used to quantitatively assess the originality of generated images without access to training data?
• RQ2 How does the number of textual inversion tokens required to reconstruct an image correlate with its conceptual novelty or originality?
• RQ3 To what extent does model familiarity with training data, as reflected in token count, predict whether an image is generic or original?
• RQ4 Can this method align with legal definitions of originality that emphasize minimal creativity and authorship?
• RQ5 How robust is the token-count-based originality metric across diverse image types and model architectures?

Key findings

• Images requiring more textual inversion tokens for reconstruction are deemed more original, as they represent novel or less familiar concepts to the model.
• Common images, such as Van Gogh’s 'Starry Night,' can be reconstructed with a single token, indicating high model familiarity and low originality.
• The method successfully correlates token count with originality across both synthetic and real-world image datasets, demonstrating consistent performance.
• Stable diffusion models generalize well to unseen elements when prompted with diverse, varied descriptions, supporting the model’s capacity for innovation.
• The framework provides a data-free, prompt-agnostic method to assess originality, making it suitable for legal and auditing applications.
• The results challenge the conventional view that memorization should be minimized, suggesting that broad model familiarity enhances originality generation when balanced with diversity.

Figure 2 : Generalization experiments diagram on synthetic data. (i) We evaluate the relationship between data diversity and originality by running experiments over sets of distinct elements in increasing sizes. (ii) Examples of datasets synthesized from the respective element sets illustrate the va