[Paper Review] Drawing out of Distribution with Neuro-Symbolic Generative Models

Yichao Liang, Joshua B. Tenenbaum+2 more|arXiv (Cornell University)|Jun 3, 2022

Handwritten Text Recognition Techniques4 citations

TL;DR

DooD is a neuro-symbolic generative model that learns general-purpose, interpretable representations of stroke-based drawings directly from raw images, using a symbolic stroke model with differentiable rendering and amortized inference. It achieves state-of-the-art zero-shot generalization across datasets and competitive performance in one-shot classification and generation tasks without supervision or test-time inference.

ABSTRACT

Learning general-purpose representations from perceptual inputs is a hallmark of human intelligence. For example, people can write out numbers or characters, or even draw doodles, by characterizing these tasks as different instantiations of the same generic underlying process -- compositional arrangements of different forms of pen strokes. Crucially, learning to do one task, say writing, implies reasonable competence at another, say drawing, on account of this shared process. We present Drawing out of Distribution (DooD), a neuro-symbolic generative model of stroke-based drawing that can learn such general-purpose representations. In contrast to prior work, DooD operates directly on images, requires no supervision or expensive test-time inference, and performs unsupervised amortised inference with a symbolic stroke model that better enables both interpretability and generalization. We evaluate DooD on its ability to generalise across both data and tasks. We first perform zero-shot transfer from one dataset (e.g. MNIST) to another (e.g. Quickdraw), across five different datasets, and show that DooD clearly outperforms different baselines. An analysis of the learnt representations further highlights the benefits of adopting a symbolic stroke model. We then adopt a subset of the Omniglot challenge tasks, and evaluate its ability to generate new exemplars (both unconditionally and conditionally), and perform one-shot classification, showing that DooD matches the state of the art. Taken together, we demonstrate that DooD does indeed capture general-purpose representations across both data and task, and takes a further step towards building general and robust concept-learning systems.

Motivation & Objective

• To develop a general-purpose, interpretable representation of stroke-based drawings that enables transfer across data and tasks.
• To overcome limitations of purely neural or symbolic models by combining differentiable rendering with symbolic stroke composition.
• To enable unsupervised, amortized inference directly from images without supervision or expensive test-time optimization.
• To demonstrate strong zero-shot generalization across diverse datasets like MNIST, QuickDraw, and Omniglot.
• To achieve competitive performance in few-shot classification and unconditional/conditional generation using a single unified model.

Proposed method

• The model uses a generative process that sequentially samples stroke locations (lt), stroke parameters (st), and stop signals (ot) via autoregressive RNNs.
• Each stroke is rendered using a differentiable spline-based renderer (δR) and composited onto a running canvas (x<t) via alpha compositing.
• A recognition model performs amortized inference by conditioning on image residuals (∆xt = x − x<t) to predict next drawing steps.
• The model employs a variational inference framework with an evidence lower bound (ELBO) objective, trained using the NVIL control variate for discrete stop variables.
• A hierarchical type-token structure enables generation of new exemplars and one-shot classification in Omniglot.
• Spatial Transformer Networks (STNs) enable differentiable spatial transformations for precise stroke placement and rendering.

Experimental results

Research questions

• RQ1 Can a neuro-symbolic model trained on one dataset (e.g., MNIST) generalize to out-of-distribution datasets (e.g., QuickDraw, Omniglot) without fine-tuning?
• RQ2 How well does the symbolic stroke model support interpretability and generalization compared to purely neural alternatives?
• RQ3 Can the model perform zero-shot transfer across datasets while maintaining high generation and reconstruction quality?
• RQ4 Does the model achieve competitive performance in few-shot classification and unconditional/conditional generation without supervision or data augmentation?
• RQ5 How does the use of amortized inference with a symbolic stroke model compare to MCMC-based inference in terms of efficiency and accuracy?

Key findings

• DooD significantly outperforms baselines in zero-shot transfer across five datasets, including MNIST to QuickDraw, demonstrating strong data generalization.
• The model achieves state-of-the-art performance in one-shot classification on Omniglot, matching or exceeding prior neuro-symbolic models without requiring additional supervision or data augmentation.
• DooD generates realistic, sharp, and artifact-free images in both unconditional and conditional generation, outperforming deep generative models that often produce blurry outputs.
• The symbolic stroke model enables interpretable disentanglement of shape components, with qualitative analysis showing meaningful compositional structure in the latent space.
• Amortized inference via the recognition model enables fast, scalable inference without expensive MCMC sampling, unlike prior symbolic models.
• The model generalizes effectively from MNIST to Omniglot, reconstructing complex characters with high fidelity through step-by-step stroke composition.