[Paper Review] FABRIC: Personalizing Diffusion Models with Iterative Feedback

Dimitri von Rütte, Elisabetta Fedele+2 more|arXiv (Cornell University)|Jul 19, 2023

Generative Adversarial Networks and Image SynthesisComputer Science3 citations

TL;DR

FABRIC is a training-free method that enhances text-to-image diffusion models by iteratively incorporating user feedback through attention-based conditioning on reference images. It improves generation quality and alignment with user preferences over multiple feedback rounds, outperforming even fine-tuned models like HPS LoRA without retraining.

ABSTRACT

In an era where visual content generation is increasingly driven by machine learning, the integration of human feedback into generative models presents significant opportunities for enhancing user experience and output quality. This study explores strategies for incorporating iterative human feedback into the generative process of diffusion-based text-to-image models. We propose FABRIC, a training-free approach applicable to a wide range of popular diffusion models, which exploits the self-attention layer present in the most widely used architectures to condition the diffusion process on a set of feedback images. To ensure a rigorous assessment of our approach, we introduce a comprehensive evaluation methodology, offering a robust mechanism to quantify the performance of generative visual models that integrate human feedback. We show that generation results improve over multiple rounds of iterative feedback through exhaustive analysis, implicitly optimizing arbitrary user preferences. The potential applications of these findings extend to fields such as personalized content creation and customization.

Motivation & Objective

• To address the challenge of personalizing text-to-image diffusion models beyond prompt engineering by integrating iterative human feedback.
• To develop a method that enhances user control and output quality in image generation without requiring model fine-tuning.
• To establish a robust evaluation framework for measuring performance improvements across multiple feedback rounds.
• To explore the trade-off between exploration and exploitation in feedback-driven generation.
• To enable orthogonal integration with existing diffusion model extensions like LoRA and checkpoints.

Proposed method

• FABRIC uses attention-based reference image conditioning to steer the diffusion process using positive and negative feedback images from prior generations.
• It modifies the self-attention mechanism in diffusion models to attend to feedback images, effectively conditioning generation on user preferences.
• The approach operates without any model retraining, making it compatible with a wide range of pre-trained diffusion models.
• Feedback is collected iteratively, with users selecting preferred or disliked outputs to guide subsequent generations.
• The method supports both automatic evaluation protocols and integration with external image corpora for feedback retrieval.
• It enables Bayesian optimization over feedback parameters to further refine generation outcomes.

Figure 2: Illustration of the proposed approach. FABRIC improves generated results by incorporating user feedback through an attention-based conditioning mechanism.

Experimental results

Research questions

• RQ1 Can iterative human feedback improve the quality and alignment of text-to-image generation in diffusion models without retraining?
• RQ2 How does feedback-based conditioning via attention mechanisms affect the diversity and distribution of generated images?
• RQ3 To what extent can FABRIC outperform fine-tuned models like HPS LoRA in preference-based generation?
• RQ4 What are the trade-offs between exploration and exploitation in feedback-driven image generation?
• RQ5 How can feedback be effectively integrated with existing diffusion model extensions such as LoRA or checkpoints?

Key findings

• FABRIC significantly improves image generation quality and user preference alignment over multiple feedback rounds, even without any training or hyperparameter tuning.
• The method outperforms the HPS LoRA, a model explicitly fine-tuned for human preference, on the relevant evaluation metric.
• Feedback leads to implicit optimization of arbitrary user objectives, such as similarity to a target image or preference for specific styles.
• Despite improvements, FABRIC tends to collapse the generation distribution to a single mode near feedback images, indicating a trade-off between exploitation and diversity.
• The approach is orthogonal to existing methods like LoRA and checkpoints, enabling additive improvements when combined with them.
• Prompt dropout is explored as a potential remedy for diversity collapse, though it risks altering the semantic content of the prompt.

(a) Highest PickScore of a generated image over all previous rounds.