[Paper Review] DreamTuner: Single Image is Enough for Subject-Driven Generation

Hua Miao, Jiawei Liu+4 more|arXiv (Cornell University)|Dec 21, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

DreamTuner proposes a training-free, subject-driven image generation method that achieves high-fidelity subject identity preservation using only a single reference image. It combines a subject-encoder for coarse identity injection and self-subject-attention layers to refine fine details, outperforming existing fine-tuning and encoder-based methods in both identity fidelity and generation quality with minimal training steps.

ABSTRACT

Diffusion-based models have demonstrated impressive capabilities for text-to-image generation and are expected for personalized applications of subject-driven generation, which require the generation of customized concepts with one or a few reference images. However, existing methods based on fine-tuning fail to balance the trade-off between subject learning and the maintenance of the generation capabilities of pretrained models. Moreover, other methods that utilize additional image encoders tend to lose important details of the subject due to encoding compression. To address these challenges, we propose DreamTurner, a novel method that injects reference information from coarse to fine to achieve subject-driven image generation more effectively. DreamTurner introduces a subject-encoder for coarse subject identity preservation, where the compressed general subject features are introduced through an attention layer before visual-text cross-attention. We then modify the self-attention layers within pretrained text-to-image models to self-subject-attention layers to refine the details of the target subject. The generated image queries detailed features from both the reference image and itself in self-subject-attention. It is worth emphasizing that self-subject-attention is an effective, elegant, and training-free method for maintaining the detailed features of customized subjects and can serve as a plug-and-play solution during inference. Finally, with additional subject-driven fine-tuning, DreamTurner achieves remarkable performance in subject-driven image generation, which can be controlled by a text or other conditions such as pose. For further details, please visit the project page at https://dreamtuner-diffusion.github.io/.

Motivation & Objective

• To address the challenge of balancing identity preservation and generation capability in subject-driven image generation with only one reference image.
• To overcome limitations of fine-tuning methods that degrade pretrained model capabilities and encoder-based methods that lose subject details due to compression.
• To develop a training-free, plug-and-play mechanism for preserving fine-grained subject details during inference.
• To enable flexible, condition-controlled generation (e.g., text or pose) of personalized subjects with minimal user input.

Proposed method

• A subject-encoder is trained to compress reference image features using a frozen CLIP image encoder and inject them via an attention layer before visual-text cross-attention for coarse identity preservation.
• Content and layout are decoupled in the subject-encoder using a pretrained depth-based ControlNet to prevent layout overfitting.
• Self-subject-attention layers are introduced by modifying original self-attention mechanisms to query features from both the generated image and the reference image, enabling iterative refinement of subject details.
• The method uses a two-stage training process: first, subject-encoder training with ControlNet; second, subject-driven fine-tuning with self-subject-attention to enhance detail fidelity.
• The self-subject-attention mechanism is applied during inference without additional training, making it a plug-and-play solution for high-fidelity subject generation.
• Hyperparameters β and ω_ref control the strength of subject-encoder injection and self-subject-attention, respectively, with optimal values found at β=0.2 and ω_ref=10.0.

Experimental results

Research questions

• RQ1 Can a single reference image suffice for high-fidelity, subject-driven image generation without compromising the pretrained model's generation capabilities?
• RQ2 How can fine-grained subject details be preserved during diffusion generation without relying on additional image encoders or extensive fine-tuning?
• RQ3 To what extent can self-attention mechanisms be adapted to preserve subject identity by querying both the generated image and the reference image?
• RQ4 What is the optimal balance between coarse identity injection and fine detail refinement in a single-image subject generation framework?

Key findings

• DreamTuner outperforms DreamBooth, Textual Inversion, ELITE, and MasaCtrl in subject fidelity (CLIP-I), achieving the highest average cosine similarity between generated images and reference images.
• The method achieves superior text consistency (CLIP-T) while maintaining high subject fidelity, indicating effective balance between prompt adherence and identity preservation.
• Abllation studies show that β=0.2 provides the best trade-off between identity consistency and editability, while ω_ref=10.0 maximizes detail refinement without quality degradation.
• Removing the ControlNet during subject-encoder training leads to layout misalignment, confirming its role in decoupling content and layout for better generalization.
• Using self-subject-attention without masking the reference image background harms generation quality, demonstrating the importance of spatially aware attention mechanisms.
• The self-subject-attention mechanism is effective as a training-free, plug-and-play component that significantly improves detail fidelity during inference.