[Paper Review] A Neural Representation of Sketch Drawings

David Ha, Douglas Eck|arXiv (Cornell University)|Apr 11, 2017

Human Motion and Animation545 citations

TL;DR

Sketch-rnn is an RNN-based variational autoencoder that learns to generate and reconstruct vector sketches, enabling conditional and unconditional sketch synthesis and latent-space exploration.

ABSTRACT

We present sketch-rnn, a recurrent neural network (RNN) able to construct stroke-based drawings of common objects. The model is trained on thousands of crude human-drawn images representing hundreds of classes. We outline a framework for conditional and unconditional sketch generation, and describe new robust training methods for generating coherent sketch drawings in a vector format.

Motivation & Objective

• Motivate the development of generative models for vector (stroke-based) drawings as an alternative to pixel-based image modelling.
• Propose a recurrent neural network framework (sketch-rnn) capable of unconditional and conditional sketch generation.
• Develop a robust training procedure for vector sketches and demonstrate latent-space encodings, interpolations, and completions.
• Release a large, public vector sketch dataset and an open-source implementation to spur further research.

Proposed method

• Represent sketches as sequences of pen stroke actions with 5-element data points (∆x, ∆y, p1, p2, p3).
• Use a bidirectional encoder RNN to produce a latent vector z via a variational autoencoder (VAE) framework (µ, σ).
• Decode with an autoregressive decoder RNN conditioned on z to predict the next point through a Gaussian Mixture Model (GMM) for (∆x, ∆y) and a categorical distribution for pen states (p1, p2, p3).
• Train with a reconstruction loss LR (likelihood of the data under the predicted distributions) plus KL divergence loss LKL between z and a standard Gaussian, with KL annealing.
• Control sampling randomness with a temperature τ to modulate diversity in generated sketches.
• Allow unconditional generation by using only the decoder, removing the latent-conditioning input.

Experimental results

Research questions

• RQ1 Can a sequence-to-sequence VAE be trained to model and generate coherent vector sketches across multiple object classes?
• RQ2 How does conditioning on a latent vector z influence the generation and reconstruction of sketches?
• RQ3 What is the role of the KL prior in shaping latent space and interpolation quality for sketches?
• RQ4 Can the model perform sketch completion, interpolation between sketches, and cross-class analogies in latent space?
• RQ5 How does the number of classes and model hyperparameters affect reconstruction and generation quality?

Key findings

• The model can generate and reconstruct vector sketches across multiple object classes in both conditional and unconditional modes.
• Latent space interpolation yields coherent morphing between sketches, with higher KL weight improving manifold coherence and interpolation quality.
• Conditional generation enables analogies and body-part manipulations in latent space, showing meaningful semantic directions.
• Sampling temperature τ modulates diversity, with lower τ yielding more deterministic, higher-quality reconstructions.
• The model demonstrates sketch completions, four-way latent interpolations, and cross-class transfers (e.g., cat with dog-like features) in qualitative evaluations.
• A public QuickDraw-based vector sketch dataset and an open-source implementation accompany the work.