[Paper Review] Sketch-based 3D Shape Retrieval using Convolutional Neural Networks

Fang Wang, Le Kang+1 more|arXiv (Cornell University)|Apr 14, 2015

3D Shape Modeling and AnalysisEngineering20 references129 citations

TL;DR

This paper proposes a sketch-based 3D shape retrieval method that bypasses the need for subjective 'best view' selection by using only two predefined viewpoints per 3D model and learning deep features via two Siamese Convolutional Neural Networks (CNNs) for sketches and views. The method significantly outperforms state-of-the-art approaches on three benchmark datasets by learning cross-domain similarity through a joint loss function, achieving superior performance in precision, recall, and mAP across all metrics.

ABSTRACT

Retrieving 3D models from 2D human sketches has received considerable attention in the areas of graphics, image retrieval, and computer vision. Almost always in state of the art approaches a large amount of "best views" are computed for 3D models, with the hope that the query sketch matches one of these 2D projections of 3D models using predefined features. We argue that this two stage approach (view selection -- matching) is pragmatic but also problematic because the "best views" are subjective and ambiguous, which makes the matching inputs obscure. This imprecise nature of matching further makes it challenging to choose features manually. Instead of relying on the elusive concept of "best views" and the hand-crafted features, we propose to define our views using a minimalism approach and learn features for both sketches and views. Specifically, we drastically reduce the number of views to only two predefined directions for the whole dataset. Then, we learn two Siamese Convolutional Neural Networks (CNNs), one for the views and one for the sketches. The loss function is defined on the within-domain as well as the cross-domain similarities. Our experiments on three benchmark datasets demonstrate that our method is significantly better than state of the art approaches, and outperforms them in all conventional metrics.

Motivation & Objective

• To address the instability and subjectivity of 'best view' selection in sketch-based 3D shape retrieval.
• To eliminate reliance on hand-crafted features by learning discriminative representations for both sketches and 3D model views.
• To improve cross-domain matching between 2D sketches and 3D model projections through joint feature learning and similarity alignment.
• To demonstrate that minimalistic viewpoint selection (only two directions) can outperform complex multi-view strategies when combined with deep feature learning.
• To validate the effectiveness of Siamese CNNs with domain-specific architectures for cross-domain shape retrieval.

Proposed method

• Uses only two fixed, significantly different viewpoints per 3D model, assuming upright orientation, to replace the complex 'best view' selection process.
• Employs two separate Siamese CNNs—one for sketches and one for 3D model views—each with domain-specific architecture to capture intrinsic differences.
• Defines a joint loss function that encourages high similarity within domains (sketch-sketch, view-view) and high cross-domain similarity (sketch-view) for matching.
• Learns shared feature representations in a common embedding space where Euclidean distance directly measures similarity.
• Trains the network end-to-end using weak supervision, with no need for paired sketch-view annotations during inference.
• Uses a minimalism approach to reduce computational cost and avoid overfitting to arbitrary viewpoint choices.

Experimental results

Research questions

• RQ1 Can sketch-based 3D shape retrieval be improved by eliminating the subjective and ambiguous 'best view' selection step?
• RQ2 Can deep feature learning with Siamese CNNs outperform hand-crafted features in matching sketches and 3D model projections?
• RQ3 Does using only two fixed viewpoints per 3D model still enable effective retrieval, especially when combined with learned features?
• RQ4 Can a joint loss function that aligns within-domain and cross-domain similarities lead to better generalization and retrieval performance?
• RQ5 How does the performance of the proposed method compare to state-of-the-art approaches across multiple benchmark datasets?

Key findings

• On the SHREC’13 benchmark, the proposed method achieves a mean average precision (mAP) of 0.469, significantly outperforming the second-best method (0.434) and all others listed in Table 3.
• The method shows a 10% performance gain over state-of-the-art methods at low recall on SHREC’13, with a much slower decline in precision as recall increases, indicating superior stability.
• On SHREC’14, the method achieves mAP of 0.228, outperforming the next best method by 75% (0.131), demonstrating consistent superiority across datasets.
• Within-domain sketch retrieval achieves mAP of 0.373, indicating that sketch inconsistency remains a major challenge, but the model still learns meaningful representations.
• Within-domain view retrieval achieves mAP of 0.909, showing that the learned features are highly effective for view-based retrieval, confirming the model’s robustness.
• The ablation study shows that using identical Siamese networks for both domains underperforms the proposed domain-specific architecture, validating the need for separate feature learning per domain.