[Paper Review] Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting

Ayan Kumar Bhunia, Pinaki Nath Chowdhury+4 more|arXiv (Cornell University)|Mar 25, 2021

Multimodal Machine Learning Applications64 references8 citations

TL;DR

This paper proposes a self-supervised learning framework for sketch and handwriting data by leveraging cross-modal translation between raster images and vector coordinate sequences—vectorization (image to vector) and rasterization (vector to image)—to learn powerful, generalizable representations. The method outperforms existing self-supervised and even some fully supervised baselines on sketch recognition, retrieval, and handwriting tasks, achieving 71.9% top-1 accuracy on QuickDraw with self-supervised pre-training.

ABSTRACT

Self-supervised learning has gained prominence due to its efficacy at learning powerful representations from unlabelled data that achieve excellent performance on many challenging downstream tasks. However supervision-free pre-text tasks are challenging to design and usually modality specific. Although there is a rich literature of self-supervised methods for either spatial (such as images) or temporal data (sound or text) modalities, a common pre-text task that benefits both modalities is largely missing. In this paper, we are interested in defining a self-supervised pre-text task for sketches and handwriting data. This data is uniquely characterised by its existence in dual modalities of rasterized images and vector coordinate sequences. We address and exploit this dual representation by proposing two novel cross-modal translation pre-text tasks for self-supervised feature learning: Vectorization and Rasterization. Vectorization learns to map image space to vector coordinates and rasterization maps vector coordinates to image space. We show that the our learned encoder modules benefit both raster-based and vector-based downstream approaches to analysing hand-drawn data. Empirical evidence shows that our novel pre-text tasks surpass existing single and multi-modal self-supervision methods.

Motivation & Objective

• To address the data annotation bottleneck in sketch and handwriting analysis by enabling self-supervised representation learning.
• To exploit the dual raster/vector nature of sketch and handwriting data as a shared supervisory signal.
• To design a unified pre-text task that benefits both image-based and sequence-based downstream models.
• To improve performance in low-data regimes where supervised learning is impractical.
• To enable cross-modality and cross-task transfer learning between sketches and handwriting.

Proposed method

• Proposes two novel cross-modal pre-text tasks: vectorization (mapping raster images to vector sequences) and rasterization (mapping vector sequences to raster images).
• Uses a shared encoder backbone (e.g., ResNet) to extract features from both modalities, with separate decoders for each direction of translation.
• Employs contrastive learning to align features from the same sketch in different modalities, encouraging disentangled and semantically rich representations.
• Applies standard data augmentations (horizontal flipping, random cropping) to improve robustness, though results are stable across augmentation strategies.
• Evaluates features via linear probing on frozen features, using a linear classifier on top of the learned representations.
• Combines features from both modalities via concatenation to further improve performance on downstream tasks.

Experimental results

Research questions

• RQ1 Can cross-modal translation between raster images and vector sequences serve as an effective self-supervised pre-text task for sketch and handwriting data?
• RQ2 Does the proposed method learn representations that generalize across different sketch categories and datasets?
• RQ3 Can the learned features from one modality (e.g., image) be effectively transferred to another (e.g., vector) for downstream tasks?
• RQ4 How does the method compare to existing self-supervised methods like SimCLR, MoCo, and CPC in terms of performance on sketch and handwriting recognition?
• RQ5 Can the self-supervised features match or surpass fully supervised baselines, especially in low-data regimes?

Key findings

• The proposed method achieves 65.1% top-1 accuracy on sketch images and 58.4% on vectors when evaluated on 80 unseen QuickDraw classes, significantly outperforming SimCLR (53.6% on images).
• On cross-dataset generalization from QuickDraw to TU-Berlin, the method achieves 58.9% top-1 accuracy on sketch images and 36.9% on vectors, demonstrating strong transferability.
• For cross-task generalization, the model achieves 37.6% top-1 accuracy on sketch images and 33.7% on vectors when evaluated on handwritten data, surpassing CPC and random baselines.
• Jointly fusing features from both image and vector modalities improves top-1 accuracy to 72.8%, exceeding the 71.9% achieved by using raster images alone.
• The method generalizes well across backbone architectures, achieving 63.3% top-1 accuracy with AlexNet and 71.9% with ResNet50, confirming design robustness.
• The model surpasses fully supervised baselines in low-data regimes and approaches their performance even with self-supervised pre-training only.