[Paper Review] SketchDreamer: Interactive Text-Augmented Creative Sketch Ideation

Zhiyu Qu, Tao Xiang+1 more|arXiv (Cornell University)|Aug 27, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

SketchDreamer introduces an interactive, text-augmented sketch generation framework that uses a text-conditioned diffusion model with a differentiable Bézier rasterizer to iteratively refine sketches based on natural language prompts. It enables non-experts to co-create storyboards through controlled, iterative sketch and text modifications, achieving superior user-perceived consistency with initial sketches compared to baselines.

ABSTRACT

Artificial Intelligence Generated Content (AIGC) has shown remarkable progress in generating realistic images. However, in this paper, we take a step "backward" and address AIGC for the most rudimentary visual modality of human sketches. Our objective is on the creative nature of sketches, and that creative sketching should take the form of an interactive process. We further enable text to drive the sketch ideation process, allowing creativity to be freely defined, while simultaneously tackling the challenge of "I can't sketch". We present a method to generate controlled sketches using a text-conditioned diffusion model trained on pixel representations of images. Our proposed approach, referred to as SketchDreamer, integrates a differentiable rasteriser of Bezier curves that optimises an initial input to distil abstract semantic knowledge from a pretrained diffusion model. We utilise Score Distillation Sampling to learn a sketch that aligns with a given caption, which importantly enable both text and sketch to interact with the ideation process. Our objective is to empower non-professional users to create sketches and, through a series of optimisation processes, transform a narrative into a storyboard by expanding the text prompt while making minor adjustments to the sketch input. Through this work, we hope to aspire the way we create visual content, democratise the creative process, and inspire further research in enhancing human creativity in AIGC. The code is available at \url{https://github.com/WinKawaks/SketchDreamer}.

Motivation & Objective

• To address the lack of interactivity and procedural creativity in existing AIGC systems for visual content generation.
• To democratize sketch creation by enabling non-professionals to generate expressive sketches through text prompts, overcoming the 'I can't sketch' barrier.
• To model sketch creation as an interactive, iterative process where both text and sketch inputs jointly guide ideation.
• To develop a method that maintains fidelity to initial sketches while allowing semantic expansion via prompt evolution.
• To explore how text and sketch modalities can co-evolve in a creative, procedural generation pipeline.

Proposed method

• A text-conditioned diffusion model is trained on pixel representations of images to generate sketch-like outputs conditioned on both text and initial sketch inputs.
• A differentiable rasterizer of Bézier curves enables end-to-end optimization of stroke parameters to align with text prompts.
• Score Distillation Sampling (SDS) is used to optimize the sketch by minimizing the distance between the generated sketch’s CLIP embedding and the text caption’s CLIP embedding.
• The method supports iterative refinement: users expand the text prompt and make minor sketch edits (e.g., resizing, relocating, adding strokes), with the model generating updated sketches at each step.
• Initial sketches are sourced from QuickDraw to reflect non-professional drawing quality, and the model is trained to preserve semantic and structural fidelity during refinement.
• A user study evaluates perceptual consistency, with participants selecting the sketch most similar to the initial input from a set of three generated options.

(a) Generation by single round.

Experimental results

Research questions

• RQ1 Can a text-conditioned diffusion model generate sketches that maintain fidelity to an initial sketch while being guided by evolving text prompts?
• RQ2 How does interactive, iterative refinement of both text and sketch inputs affect the perceived consistency and creativity of generated sketches?
• RQ3 To what extent can a differentiable Bézier rasterizer enable controllable, high-fidelity sketch generation from text and initial sketch inputs?
• RQ4 How does the proposed method compare to baseline approaches in preserving initial sketch structure under text-guided generation?
• RQ5 Can non-professional users effectively use the system to co-create storyboards through natural language and minimal sketch edits?

Key findings

• SketchDreamer achieved a user study preference rate of 68.34%, significantly outperforming VectorFusion (init) at 22.91% and VectorFusion at 8.75% in selecting sketches most consistent with the initial input.
• The method achieved an R-Prec score of 72.94% and a CLIP similarity score of 31.50, indicating strong alignment with text prompts while preserving initial sketch structure.
• Despite low initial sketch quality (R-Prec of 28.91% for QuickDraw), SketchDreamer significantly improved consistency, demonstrating robustness to poor-quality inputs.
• The method outperformed both uncontrollable and baseline controllable methods in perceptual consistency, suggesting effective integration of text and sketch in the optimization process.
• User study results confirm that iterative refinement with both text and sketch edits leads to more coherent and consistent storyboard generation than text-only or non-interactive methods.
• The approach effectively addresses the 'I can't sketch' problem by enabling users to generate expressive sketches through natural language and minimal manual input.

(b) Interactive generation by multiple rounds.