[Paper Review] Deep Neural Networks for Sketch Recognition.

Yongxin Yang, Timothy M. Hospedales|arXiv (Cornell University)|Jan 30, 2015

Advanced Image and Video Retrieval Techniques9 references42 citations

TL;DR

This paper introduces a multi-scale, multi-channel deep neural network specifically designed for sketch recognition, leveraging sketch-specific statistics, sequential stroke ordering, and multi-scale feature fusion to outperform humans on the largest sketch dataset to date. The model achieves state-of-the-art accuracy while remaining small and trainable on CPUs.

ABSTRACT

We propose a multi-scale multi-channel deep neural network framework that, for the first time, yields sketch recognition performance surpassing that of humans. Our superior performance is a result of explicitly embedding the unique characteristics of sketches in our model: (i) a network architecture designed for sketch rather than natural photo statistics, (ii) a multi-channel generalisation that encodes sequential ordering in the sketching process, and (iii) a multi-scale network ensemble with joint Bayesian fusion that accounts for the different levels of abstraction exhibited in free-hand sketches. We show that state-of-the-art deep networks specifically engineered for photos of natural objects fail to perform well on sketch recognition, regardless whether they are trained using photo or sketch. Our network on the other hand not only delivers the best performance on the largest human sketch dataset to date, but also is small in size making efficient training possible using just CPUs.

Motivation & Objective

• Address the gap in deep learning models for sketch recognition, as existing photo-optimized networks underperform on sketches.
• Overcome the limitations of transfer learning from photo datasets by designing a network architecture tailored to sketch-specific characteristics.
• Improve recognition accuracy by explicitly modeling stroke sequence and multi-level abstraction in free-hand sketches.
• Develop an efficient, small-scale model that enables fast training on CPUs without sacrificing performance.

Proposed method

• Design a custom deep neural network architecture optimized for sketch data rather than natural photo statistics.
• Implement a multi-channel mechanism to encode the sequential ordering of strokes in free-hand sketches.
• Construct a multi-scale network ensemble to capture features at varying levels of abstraction in sketches.
• Apply joint Bayesian fusion to combine predictions from multiple scales, enhancing robustness and accuracy.
• Train the model end-to-end on a large-scale human sketch dataset, avoiding reliance on pre-trained photo models.
• Optimize model size and training efficiency to allow effective training on CPU-only hardware.

Experimental results

Research questions

• RQ1 Can a deep neural network specifically designed for sketches outperform human-level performance on sketch recognition tasks?
• RQ2 To what extent do photo-optimized deep networks fail when applied to sketch data, even with fine-tuning on sketch data?
• RQ3 How does modeling stroke sequence and multi-scale abstraction improve sketch recognition accuracy?
• RQ4 Can a small, efficient model achieve state-of-the-art performance without requiring GPU acceleration?

Key findings

• The proposed model achieves the highest recognition accuracy on the largest human sketch dataset ever collected, surpassing human-level performance.
• Photo-optimized deep networks, even when fine-tuned on sketch data, fail to generalize well, demonstrating the need for sketch-specific architectures.
• The multi-channel design that encodes stroke sequence significantly improves recognition by capturing the temporal dynamics of sketching.
• The multi-scale ensemble with Bayesian fusion effectively combines features across abstraction levels, boosting overall accuracy.
• The model remains small and efficient, enabling effective training on CPUs, which is a significant advantage over large GPU-dependent models.
• The framework sets a new state of the art in sketch recognition, demonstrating that architecture specialization for sketch data is critical for performance.