[Paper Review] IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models

Ye Hu, Jun Zhang+3 more|arXiv (Cornell University)|Aug 13, 2023

Generative Adversarial Networks and Image Synthesis109 citations

TL;DR

IP-Adapter introduces a lightweight, decoupled cross-attention image-prompt adapter for pretrained text-to-image diffusion models, achieving image-prompt capability with about 22M parameters and strong generalization.

ABSTRACT

Recent years have witnessed the strong power of large text-to-image diffusion models for the impressive generative capability to create high-fidelity images. However, it is very tricky to generate desired images using only text prompt as it often involves complex prompt engineering. An alternative to text prompt is image prompt, as the saying goes: "an image is worth a thousand words". Although existing methods of direct fine-tuning from pretrained models are effective, they require large computing resources and are not compatible with other base models, text prompt, and structural controls. In this paper, we present IP-Adapter, an effective and lightweight adapter to achieve image prompt capability for the pretrained text-to-image diffusion models. The key design of our IP-Adapter is decoupled cross-attention mechanism that separates cross-attention layers for text features and image features. Despite the simplicity of our method, an IP-Adapter with only 22M parameters can achieve comparable or even better performance to a fully fine-tuned image prompt model. As we freeze the pretrained diffusion model, the proposed IP-Adapter can be generalized not only to other custom models fine-tuned from the same base model, but also to controllable generation using existing controllable tools. With the benefit of the decoupled cross-attention strategy, the image prompt can also work well with the text prompt to achieve multimodal image generation. The project page is available at \url{https://ip-adapter.github.io}.

Motivation & Objective

• Enable image prompt generation without fine-tuning the base diffusion model.
• Design a lightweight adapter that preserves text-to-image capabilities while incorporating image prompts.
• Achieve strong generalization to custom models and compatibility with controllable tools like ControlNet.
• Demonstrate multimodal prompting by combining image prompts with text prompts.

Proposed method

• Add an image encoder (CLIP image encoder) to produce a global image embedding.
• Introduce decoupled cross-attention: add a new cross-attention layer for image features at each UNet cross-attention, with trainable K/V projections.
• Initialize image-K/V projections from text-K/V to speed convergence and train only the adapter parameters (~22M total).
• Train with the same diffusion objective L_simple, conditioning on text c_t and image c_i; randomly drop image/text prompts to enable classifier-free guidance.
• During inference, optionally weight image vs. text guidance with a parameter lambda to balance multimodal prompts.
• Demonstrate compatibility with existing controllable adapters (e.g., ControlNet) without modifying the base diffusion model.

Experimental results

Research questions

• RQ1 Can image prompts be integrated into pretrained text-to-image diffusion models without fine-tuning the base model?
• RQ2 Does a decoupled cross-attention design provide better image-prompt fidelity than simple feature concatenation or other adapters?
• RQ3 Is the IP-Adapter reusable across custom models derived from the same base model and compatible with existing control tools?
• RQ4 Can image prompts be effectively combined with text prompts to enable multimodal generation?

Key findings

• A 22M-parameter IP-Adapter achieves comparable or better results than some fully fine-tuned image-prompt models.
• The decoupled cross-attention design outperforms simple adapters that concatenate image features into cross-attention.
• IP-Adapter is reusable on custom models derived from the same base model and remains compatible with controllable tools like ControlNet.
• The method supports multimodal prompts, enabling balanced use of image and text conditions for generation.
• Quantitative results on COCO show IP-Adapter surpasses several adapters and matches or exceeds some fine-tuned baselines (CLIP-T and CLIP-I metrics).