[Paper Review] SketchMate: Deep Hashing for Million-Scale Human Sketch Retrieval

Peng Xu, Yongye Huang+7 more|arXiv (Cornell University)|Apr 4, 2018

Advanced Image and Video Retrieval Techniques22 references16 citations

TL;DR

SketchMate proposes a deep hashing framework for million-scale human sketch retrieval, leveraging a novel two-branch CNN-RNN architecture to model stroke order and a sketch-specific hashing loss to handle abstraction. It achieves state-of-the-art performance on retrieval, zero-shot generalization, and sketch recognition, outperforming prior methods on a 3.8M-sketch dataset.

ABSTRACT

We propose a deep hashing framework for sketch retrieval that, for the first time, works on a multi-million scale human sketch dataset. Leveraging on this large dataset, we explore a few sketch-specific traits that were otherwise under-studied in prior literature. Instead of following the conventional sketch recognition task, we introduce the novel problem of sketch hashing retrieval which is not only more challenging, but also offers a better testbed for large-scale sketch analysis, since: (i) more fine-grained sketch feature learning is required to accommodate the large variations in style and abstraction, and (ii) a compact binary code needs to be learned at the same time to enable efficient retrieval. Key to our network design is the embedding of unique characteristics of human sketch, where (i) a two-branch CNN-RNN architecture is adapted to explore the temporal ordering of strokes, and (ii) a novel hashing loss is specifically designed to accommodate both the temporal and abstract traits of sketches. By working with a 3.8M sketch dataset, we show that state-of-the-art hashing models specifically engineered for static images fail to perform well on temporal sketch data. Our network on the other hand not only offers the best retrieval performance on various code sizes, but also yields the best generalization performance under a zero-shot setting and when re-purposed for sketch recognition. Such superior performances effectively demonstrate the benefit of our sketch-specific design.

Motivation & Objective

• To address the lack of large-scale, diverse human sketch datasets in sketch retrieval research.
• To introduce sketch hashing retrieval (SHR) as a novel, more challenging benchmark than sketch recognition, requiring both fine-grained feature learning and compact binary codes.
• To model the sequential, dynamic nature of human sketching by exploiting stroke ordering through a two-branch CNN-RNN architecture.
• To design a sketch-specific hashing loss that enforces compact, discriminative clusters in Hamming space, accommodating high abstraction levels.
• To validate the generalization of the proposed framework to sketch recognition and zero-shot retrieval.

Proposed method

• Constructs a 3.8M human sketch dataset, QuickDraw-3.8M, by sampling from the Google QuickDraw dataset to enable large-scale analysis.
• Employs a two-branch CNN-RNN architecture: one branch processes raw sketch images as static features, the other models stroke order via RNNs on sequential pixelmaps.
• Introduces a novel sketch-specific hashing loss that combines contrastive loss, center loss (SCL), and quantization loss to enhance intra-class compactness and inter-class separation in Hamming space.
• Uses a joint embedding layer to fuse CNN and RNN features before projecting to binary hash codes via a differentiable binarization function.
• Optimizes the end-to-end network using a multi-component loss: $\mathcal{L} = \mathcal{L}_{\text{contrastive}} + \mathcal{L}_{\text{SCL}} + \mathcal{L}_{\text{quantization}}$.
• Evaluates performance under varying code lengths (16–64 bits), zero-shot settings, and downstream sketch recognition tasks.

Experimental results

Research questions

• RQ1 Can a deep hashing framework effectively scale to million-scale human sketch retrieval while preserving temporal stroke order?
• RQ2 How does modeling stroke sequence via RNN improve feature representation for sketch retrieval compared to static CNNs?
• RQ3 Does a sketch-specific hashing loss that accounts for abstraction and clustering outperform standard hashing losses on large-scale sketch data?
• RQ4 Can the proposed model generalize to zero-shot sketch retrieval and sketch recognition tasks?
• RQ5 How does the model’s performance compare to state-of-the-art methods on both retrieval and recognition benchmarks?

Key findings

• The proposed SketchMate model achieves a mean average precision (MAP) of 0.7547 on zero-shot sketch hashing retrieval, significantly outperforming DLBHC (0.7094) and DSH-Sketch (0.5334).
• With 64-bit hash codes, SketchMate achieves a precision of 0.8051 on sketch recognition, surpassing Sketch-a-Net (0.6871) and ResNet-50 (0.7864).
• The addition of sketch-specific center loss (SCL) improves performance by 1.02% over standard center loss, which degrades performance, validating the necessity of sketch-aware design.
• The model achieves real-time retrieval on a 3.8M sketch gallery, with a query time of under 0.1 seconds and a memory load of only 12.5 MB.
• Qualitative results show fewer semantic errors and more plausible false positives, indicating better feature disentanglement and robustness.
• The two-branch CNN-RNN architecture with temporal modeling yields 79.49% accuracy on sketch recognition, outperforming the single-branch RNN (77.88%) and CNN (73.76%) variants.