[Paper Review] CLIPstyler: Image Style Transfer with a Single Text Condition

Gihyun Kwon, Jong Chul Ye|arXiv (Cornell University)|Dec 1, 2021

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

CLIPstyler proposes a novel image style transfer method that generates realistic style transfers using only a single text description, eliminating the need for reference style images. By leveraging CLIP's text-image embeddings and a patch-wise contrastive loss with multiview augmentations, the method enables fine-grained, semantically grounded style transfer that preserves content structure while producing vivid, diverse textures aligned with textual prompts.

ABSTRACT

Existing neural style transfer methods require reference style images to transfer texture information of style images to content images. However, in many practical situations, users may not have reference style images but still be interested in transferring styles by just imagining them. In order to deal with such applications, we propose a new framework that enables a style transfer `without' a style image, but only with a text description of the desired style. Using the pre-trained text-image embedding model of CLIP, we demonstrate the modulation of the style of content images only with a single text condition. Specifically, we propose a patch-wise text-image matching loss with multiview augmentations for realistic texture transfer. Extensive experimental results confirmed the successful image style transfer with realistic textures that reflect semantic query texts.

Motivation & Objective

• To enable image style transfer without requiring reference style images, addressing a key limitation in existing neural style transfer methods.
• To transfer realistic and diverse textures to content images based solely on natural language descriptions of desired styles.
• To overcome the reliance on pre-trained generative models or instance normalization layers (e.g., AdaIN) by training a lightweight CNN for text-conditioned stylization.
• To improve semantic alignment between text prompts and visual textures through a patch-wise CLIP-based loss with multiview augmentation.
• To reduce over-stylization artifacts via a novel threshold regularization on high-scoring patches during training.

Proposed method

• A lightweight CNN is trained to transform content images based on a single text condition, using CLIP's text-image embedding space as a supervisory signal.
• A patch-wise contrastive loss is introduced, where patches from the generated image are compared to the text embedding via CLIP, encouraging local texture alignment.
• Multiview data augmentations (e.g., rotated or cropped patches) are applied to increase diversity and realism in transferred textures.
• A threshold regularization is applied to suppress patches with abnormally high similarity scores, preventing overfitting to specific patch patterns.
• The method supports both optimization-based and fast inference-based inference modes, enabling high-resolution stylization.
• The model is trained per text condition, allowing flexible style transfer across diverse textual prompts.

Figure 2 : Overall schematics of our proposed patch-wise CLIP loss. We optimize the neural network $f$ using the loss functions.

Experimental results

Research questions

• RQ1 Can image style transfer be effectively performed using only a single text description, without any reference style image?
• RQ2 Can CLIP's pre-trained text-image embeddings be leveraged to produce semantically meaningful and realistic style transfers?
• RQ3 Does a patch-wise contrastive loss with multiview augmentation improve local texture realism and diversity in style transfer?
• RQ4 Can threshold regularization mitigate over-stylization artifacts caused by high-similarity patches during training?
• RQ5 How does the method compare to existing CLIP-based or AdaIN-based style transfer approaches in terms of content preservation and style accuracy?

Key findings

• The proposed method successfully transfers realistic and diverse textures to content images based solely on text descriptions, as demonstrated in qualitative results across various styles such as 'watercolor', 'fauvism', and 'ink wash painting'.
• The patch-wise CLIP loss with multiview augmentation significantly improves texture realism and diversity, outperforming baseline methods in visual quality and semantic alignment.
• The threshold regularization effectively reduces over-stylization artifacts caused by high-scoring patches, improving overall visual consistency.
• High-resolution outputs (up to 1920×2580) demonstrate the method’s capability to generate detailed, stylized images suitable for practical applications.
• User study results and qualitative comparisons show that CLIPstyler outperforms existing methods like AdaIN+CLIP, StyleCLIP, and VQGAN+CLIP in both content preservation and style synthesis quality.
• Failure cases reveal limitations related to poor patch sampling and over-reliance on literal text visualization, suggesting future improvements in sampling strategy and prompt engineering.

Figure 3 : Style transfer results on various query text conditions. Our method can synthesize realistic textures which reflect the text conditions. Additional results are in our Supplementary Materials.