[Paper Review] Zero-shot Image-to-Image Translation

Gaurav Parmar, Krishna Kumar Singh+4 more|arXiv (Cornell University)|Feb 6, 2023

Generative Adversarial Networks and Image SynthesisComputer Science3 citations

TL;DR

This paper proposes pix2pix-zero, a training-free, prompt-free image-to-image translation method that enables zero-shot editing of real images using pre-trained text-to-image diffusion models. By automatically discovering edit directions in CLIP embedding space and applying cross-attention guidance to preserve input structure, the method achieves state-of-the-art performance in content preservation and photorealism without fine-tuning or manual prompting.

ABSTRACT

Large-scale text-to-image generative models have shown their remarkable ability to synthesize diverse and high-quality images. However, it is still challenging to directly apply these models for editing real images for two reasons. First, it is hard for users to come up with a perfect text prompt that accurately describes every visual detail in the input image. Second, while existing models can introduce desirable changes in certain regions, they often dramatically alter the input content and introduce unexpected changes in unwanted regions. In this work, we propose pix2pix-zero, an image-to-image translation method that can preserve the content of the original image without manual prompting. We first automatically discover editing directions that reflect desired edits in the text embedding space. To preserve the general content structure after editing, we further propose cross-attention guidance, which aims to retain the cross-attention maps of the input image throughout the diffusion process. In addition, our method does not need additional training for these edits and can directly use the existing pre-trained text-to-image diffusion model. We conduct extensive experiments and show that our method outperforms existing and concurrent works for both real and synthetic image editing.

Motivation & Objective

• To address the challenge of editing real images with pre-trained text-to-image diffusion models, which require cumbersome manual prompting and often fail to preserve input structure.
• To eliminate the need for task-specific fine-tuning or image-specific text prompts in image-to-image translation.
• To enable zero-shot editing of diverse real and synthetic images while preserving layout, object pose, and background details.
• To develop a method that leverages existing pre-trained models without additional training or costly inversion processes.

Proposed method

• Automatically discover edit directions in CLIP embedding space by computing the mean difference between sentence embeddings of source and target domain terms (e.g., 'cat' vs. 'dog').
• Use deterministic DDIM inversion with autocorrelation regularization to ensure noise remains close to Gaussian, improving inversion quality and stability.
• Apply cross-attention guidance during diffusion sampling to preserve the cross-attention maps of the input image, maintaining structural consistency.
• Leverage the pre-trained text-to-image diffusion model directly without fine-tuning, enabling zero-shot adaptation to new edit tasks.
• Distill the high-fidelity diffusion model outputs into a fast conditional GAN using paired real-edited image data for real-time inference.

Experimental results

Research questions

• RQ1 Can edit directions be automatically discovered in CLIP embedding space without requiring manual text prompts for each input image?
• RQ2 Can cross-attention map consistency during diffusion preserve the structural layout of the input image during editing?
• RQ3 Can a training-free, prompt-free method achieve superior content preservation and photorealism compared to existing zero-shot and fine-tuned approaches?
• RQ4 How effective is the combination of sentence-based edit direction discovery and cross-attention guidance in maintaining object pose and background fidelity?

Key findings

• The proposed method achieves the highest CLIP-Accuracy (0.87) while maintaining the lowest Structure Distance (0.04) and BG LPIPS (0.05), indicating superior edit consistency and content preservation.
• Cross-attention guidance significantly improves structure retention, reducing structural distortion and background error compared to ablated variants without this component.
• The method outperforms SDEdit and DDIM + word swap in both photorealism and content preservation, especially in complex editing tasks like object replacement and style transfer.
• With conditional GAN distillation, inference speed is accelerated by approximately 3,800×, enabling real-time editing on consumer hardware.
• The method is robust across diverse real and synthetic images, including object replacement (cat → dog), attribute editing (adding glasses), and style transfer (sketch → oil pastel).