[Paper Review] DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation

Gwanghyun Kim, Kwon, Taesung+1 more|arXiv (Cornell University)|Oct 6, 2021

Image Processing Techniques and Applications38 citations

TL;DR

DiffusionCLIP uses diffusion models fine-tuned with CLIP guidance to enable robust, zero-shot text-driven image manipulation, including unseen domains and multi-attribute changes.

ABSTRACT

Recently, GAN inversion methods combined with Contrastive Language-Image Pretraining (CLIP) enables zero-shot image manipulation guided by text prompts. However, their applications to diverse real images are still difficult due to the limited GAN inversion capability. Specifically, these approaches often have difficulties in reconstructing images with novel poses, views, and highly variable contents compared to the training data, altering object identity, or producing unwanted image artifacts. To mitigate these problems and enable faithful manipulation of real images, we propose a novel method, dubbed DiffusionCLIP, that performs text-driven image manipulation using diffusion models. Based on full inversion capability and high-quality image generation power of recent diffusion models, our method performs zero-shot image manipulation successfully even between unseen domains and takes another step towards general application by manipulating images from a widely varying ImageNet dataset. Furthermore, we propose a novel noise combination method that allows straightforward multi-attribute manipulation. Extensive experiments and human evaluation confirmed robust and superior manipulation performance of our methods compared to the existing baselines. Code is available at https://github.com/gwang-kim/DiffusionCLIP.git.

Motivation & Objective

• Motivate robust zero-shot image manipulation beyond GAN inversion limitations on diverse real images.
• Leverage the inversion and generation power of diffusion models to faithfully edit content while preserving identity.
• Enable manipulation across unseen domains and enable translation between unseen domains.
• Introduce a noise-combination approach to perform multi-attribute manipulation within a single sampling process.

Proposed method

• Use a pretrained diffusion model to map the input image to latent noise via a forward diffusion (DDIM/ODE-based).
• Fine-tune the reverse diffusion model using CLIP-guided losses to steer attributes toward a target text while preserving identity.
• Adopt the directional CLIP loss to align the image and text direction in CLIP space, supplemented by an identity loss to prevent undesired changes.
• Utilize deterministic forward and reverse DDIM sampling for near-perfect inversion and controlled generation.
• Introduce a fast sampling strategy with return steps and reduced forward/generation steps to balance quality and speed.
• Enable multi-attribute transfer by linearly combining noises from multiple fine-tuned models during sampling.

Experimental results

Research questions

• RQ1 Can diffusion-based inversion faithfully manipulate real images in-domain and out-of-domain with text prompts?
• RQ2 Can the approach translate between unseen domains and synthesize unseen-domain images from strokes or other inputs?
• RQ3 Does combining noises from multiple fine-tuned models enable multi-attribute manipulation in a single sampling process?
• RQ4 What are the optimal sampling hyperparameters to balance reconstruction quality, speed, and attribute control?

Key findings

• DiffusionCLIP achieves near-perfect reconstruction quality surpassing GAN-inversion baselines across MAE, LPIPS, and SSIM metrics.
• It enables manipulation of real images into unseen domains and translation between unseen domains, outperforming baselines in qualitative and human evaluations.
• Directional CLIP loss with identity constraints yields robust attribute control with high segmentation-consistency and identity preservation.
• Multi-attribute transfer is achievable by combining noises from multiple fine-tuned models during a single sampling step.
• A fast sampling regime with return steps and reduced step counts provides practical speedups with only modest loss in fidelity.