[Paper Review] SketchParse : Towards Rich Descriptions for Poorly Drawn Sketches using Multi-Task Hierarchical Deep Networks

Ravi Kiran Sarvadevabhatla, Isht Dwivedi+3 more|ePrints@IISc (Indian Institute of Science)|Sep 5, 2017

Multimodal Machine Learning Applications50 references8 citations

TL;DR

SketchParse introduces a novel multi-task hierarchical deep network for fully automatic parsing of freehand sketches, leveraging sketchified photos from part-annotated datasets to bypass manual annotation. It uses shared feature extraction, category-specific experts, a router for dynamic routing, and 2D pose estimation as an auxiliary task, achieving state-of-the-art performance in part segmentation, pose estimation, and sketch captioning across multiple datasets and unseen categories.

ABSTRACT

The ability to semantically interpret hand-drawn line sketches, although very challenging, can pave way for novel applications in multimedia. We propose SketchParse, the first deep-network architecture for fully automatic parsing of freehand object sketches. SketchParse is configured as a two-level fully convolutional network. The first level contains shared layers common to all object categories. The second level contains a number of expert sub-networks. Each expert specializes in parsing sketches from object categories which contain structurally similar parts. Effectively, the two-level configuration enables our architecture to scale up efficiently as additional categories are added. We introduce a router layer which (i) relays sketch features from shared layers to the correct expert (ii) eliminates the need to manually specify object category during inference. To bypass laborious part-level annotation, we sketchify photos from semantic object-part image datasets and use them for training. Our architecture also incorporates object pose prediction as a novel auxiliary task which boosts overall performance while providing supplementary information regarding the sketch. We demonstrate SketchParse's abilities (i) on two challenging large-scale sketch datasets (ii) in parsing unseen, semantically related object categories (iii) in improving fine-grained sketch-based image retrieval. As a novel application, we also outline how SketchParse's output can be used to generate caption-style descriptions for hand-drawn sketches.

Motivation & Objective

• To enable fully automatic parsing of freehand sketches across diverse object categories with fine-grained semantic understanding.
• To overcome the challenge of sparse, low-quality sketches with high variability in drawing skill.
• To eliminate the need for costly manual part-level sketch annotations by using sketchified images from existing part-annotated photo datasets.
• To improve sketch understanding by incorporating 2D pose estimation as an auxiliary task.
• To enable novel applications such as sketch captioning and fine-grained sketch-based image retrieval.

Proposed method

• The model uses a two-level fully convolutional architecture: shared layers for all categories and expert sub-networks for structurally similar super-categories (e.g., birds and airplanes).
• A router network dynamically routes features from shared layers to the appropriate expert based on predicted super-category, eliminating manual category specification at inference.
• Training data is generated by sketchifying images from existing semantic part-detection datasets, preserving part-level annotations while transforming images into sketch-like representations.
• The network is trained end-to-end with a multi-task loss combining part segmentation and 2D pose estimation to improve generalization and performance.
• A graph-based re-ranking method uses part-level parsing and pose to improve sketch-based image retrieval by matching part graphs between sketches and images.
• Sketch descriptions are generated via a template-filling approach using outputs from the category router, part parser, and pose network.

Experimental results

Research questions

• RQ1 Can a deep hierarchical network with shared and expert sub-networks effectively parse freehand sketches across multiple object categories?
• RQ2 Can sketchified images from part-annotated photo datasets serve as a viable alternative to manually annotated sketch datasets for training?
• RQ3 Does incorporating 2D pose estimation as an auxiliary task improve the performance of sketch part segmentation?
• RQ4 Can the model generalize to unseen, semantically related object categories without retraining?
• RQ5 Can the parsed sketch features be used to generate rich, descriptive captions for hand-drawn sketches?

Key findings

• SketchParse achieves state-of-the-art performance on two large-scale sketch datasets, demonstrating strong generalization across diverse object categories.
• The model significantly improves fine-grained sketch-based image retrieval by using part-level parsing and pose information in a graph-matching re-ranking framework.
• The inclusion of 2D pose estimation as an auxiliary task boosts part segmentation accuracy, indicating effective multi-task learning.
• The router mechanism enables automatic category inference at test time, eliminating the need for manual category input.
• The model successfully generates detailed, caption-style descriptions of sketches using parsed parts, pose, and category information.
• The approach generalizes well to unseen, semantically related categories, demonstrating scalability and robustness.