[Paper Review] CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders

Kevin Frans, L. B. Soros+1 more|arXiv (Cornell University)|Jun 28, 2021

Handwritten Text Recognition Techniques16 references93 citations

TL;DR

CLIPDraw synthesizes vector-stroke drawings from text by gradient-descent optimization against a pre-trained CLIP model, without training a new model.

ABSTRACT

This work presents CLIPDraw, an algorithm that synthesizes novel drawings based on natural language input. CLIPDraw does not require any training; rather a pre-trained CLIP language-image encoder is used as a metric for maximizing similarity between the given description and a generated drawing. Crucially, CLIPDraw operates over vector strokes rather than pixel images, a constraint that biases drawings towards simpler human-recognizable shapes. Results compare between CLIPDraw and other synthesis-through-optimization methods, as well as highlight various interesting behaviors of CLIPDraw, such as satisfying ambiguous text in multiple ways, reliably producing drawings in diverse artistic styles, and scaling from simple to complex visual representations as stroke count is increased. Code for experimenting with the method is available at: https://colab.research.google.com/github/kvfrans/clipdraw/blob/main/clipdraw.ipynb

Motivation & Objective

• Demonstrate a training-free method to generate drawings from natural language prompts.
• Examine how CLIP-based optimization behaves when restricted to vector stroke representations.
• Investigate how drawing style, stroke count, and prompts affect output shapes and recognizability.
• Provide a testbed for exploring language–image relationships and AI-assisted art.

Proposed method

• Represent drawings as a fixed set of RGBA Bézier curves with differentiable rendering.
• Use a pre-trained CLIP model to encode both the prompt and the rendered drawings.
• Optimize curve control points, thickness, and colors via gradient descent to minimize negative cosine similarity between prompt and image encodings.
• Augment rendered images with random perspective shifts and crops to enforce recognizability under distortions.
• Iterate optimization for a fixed number of steps until convergence.

Experimental results

Research questions

• RQ1 How does CLIPDraw compare to other synthesis-through-optimization methods in producing recognizable drawings?
• RQ2 How do style descriptors in prompts influence the visual style of the drawings?
• RQ3 What is the effect of stroke count on realism and detail in generated drawings?
• RQ4 Can CLIPDraw reveal cultural or abstract associations through prompts?
• RQ5 How effective are negative prompts in steering CLIPDraw outputs?

Key findings

• CLIPDraw produces diverse, human-recognizable drawings using simple vector strokes.
• Style descriptors (“watercolor”, “3D rendering”) yield markedly different artistic outputs for the same prompt.
• Increasing stroke count tends to yield more detailed and realistic renderings.
• CLIPDraw can interpret ambiguous prompts in multiple ways and occasionally embeds text or symbols related to the prompt in the image.
• Negative prompts can partially steer outputs, but their effects are not consistently reliable across prompts.