[Paper Review] Conditional Score Guidance for Text-Driven Image-to-Image Translation

Hyun‐Soo Lee, Minsoo Kang+1 more|arXiv (Cornell University)|May 29, 2023

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes a training-free, conditional score guidance method for text-driven image-to-image translation using pretrained text-to-image diffusion models. By deriving a novel conditional score function that incorporates both source image and source text, along with a cross-attention mixup technique, the method enables high-fidelity editing of specific image regions while preserving irrelevant background content, outperforming prior approaches in structural fidelity and edit consistency.

ABSTRACT

We present a novel algorithm for text-driven image-to-image translation based on a pretrained text-to-image diffusion model. Our method aims to generate a target image by selectively editing the regions of interest in a source image, defined by a modifying text, while preserving the remaining parts. In contrast to existing techniques that solely rely on a target prompt, we introduce a new score function that additionally considers both the source image and the source text prompt, tailored to address specific translation tasks. To this end, we derive the conditional score function in a principled manner, decomposing it into the standard score and a guiding term for target image generation. For the gradient computation of the guiding term, we assume a Gaussian distribution of the posterior distribution and estimate its mean and variance to adjust the gradient without additional training. In addition, to improve the quality of the conditional score guidance, we incorporate a simple yet effective mixup technique, which combines two cross-attention maps derived from the source and target latents. This strategy is effective for promoting a desirable fusion of the invariant parts in the source image and the edited regions aligned with the target prompt, leading to high-fidelity target image generation. Through comprehensive experiments, we demonstrate that our approach achieves outstanding image-to-image translation performance on various tasks.

Motivation & Objective

• To address the lack of controllability in existing text-to-image diffusion models when performing image-to-image translation.
• To enable precise editing of specific image regions based on modifying text, while preserving the rest of the image.
• To develop a training-free method that enhances guidance in diffusion sampling using source image and source text context.
• To improve generation quality through a novel cross-attention mixup strategy that fuses source and target latent representations.
• To introduce a new metric for evaluating the preservation of pairwise image relationships before and after translation.

Proposed method

• Derives a conditional score function that combines the standard score conditioned on the target text with a guiding term based on the posterior of the source latent given the target latent and target prompt.
• Models the posterior distribution as Gaussian, estimating its mean and covariance without additional training to compute the guiding gradient.
• Introduces a cross-attention mixup technique that adaptively blends cross-attention maps from source and target latents at each denoising step.
• Applies the conditional score guidance during DDIM sampling to steer the diffusion process toward the desired edit while preserving structural consistency.
• Uses CLIP-encodings for text prompts and latent space representations from a pretrained diffusion model (e.g., Stable Diffusion).
• Employs a background mask to estimate the covariance matrix of the Gaussian posterior, focusing guidance on relevant image regions.

Experimental results

Research questions

• RQ1 Can a conditional score function that incorporates both source image and source text improve controllability in text-driven image-to-image translation?
• RQ2 How does the proposed guiding term based on a Gaussian posterior of the source latent enhance edit fidelity without fine-tuning?
• RQ3 To what extent does cross-attention mixup improve the fusion of invariant background and edited regions in generated images?
• RQ4 How does the method compare to existing training-free and fine-tuned approaches in preserving structural relationships between source and target images?
• RQ5 Can a novel metric measuring pairwise image relationships effectively evaluate translation fidelity?

Key findings

• The proposed method, CSG, achieves the best structural fidelity (SD) and region-wise edit consistency (RD) across all evaluated tasks, outperforming baselines including DDIM and Prompt-to-Prompt.
• On the LAION 5B dataset, CSG achieves an SD of 0.0150 and RD of 0.0192 on the 'dog → cat' task, significantly lower than DDIM (SD: 0.0748, RD: 0.4608).
• The ablation study confirms that both conditional score guidance and cross-attention mixup contribute to performance, with mixup providing a measurable improvement over CSG without it.
• Qualitative results show CSG successfully edits only the target object (e.g., cat to dog) while preserving background structure, unlike methods that alter the entire image.
• The method is more efficient than Pix2Pix-Zero, avoiding backpropagation through the noise network, thus reducing memory and inference time.
• The proposed metric for pairwise relationship fidelity shows that CSG maintains structural consistency better than existing methods, even when traditional metrics like CLIP score (CS) are lower.