[Paper Review] Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval

Aneeshan Sain, Ayan Kumar Bhunia+3 more|arXiv (Cornell University)|Jul 29, 2020

Advanced Image and Video Retrieval TechniquesComputer Science58 references22 citations

TL;DR

This paper proposes a cross-modal hierarchical modelling framework for fine-grained sketch-based image retrieval (FG-SBIR) that discovers implicit sketch hierarchies through end-to-end trainable, Gumbel-Softmax-approximated node merging. By integrating cross-modal co-attention and hierarchical fusion at each level, the method learns discriminative, multi-level embeddings that significantly outperform state-of-the-art methods on standard benchmarks, achieving 62.45% accuracy@1 on QMUL-ChairV2 and 67.23% on SWIRE.

ABSTRACT

Sketch as an image search query is an ideal alternative to text in capturing the fine-grained visual details. Prior successes on fine-grained sketch-based image retrieval (FG-SBIR) have demonstrated the importance of tackling the unique traits of sketches as opposed to photos, e.g., temporal vs. static, strokes vs. pixels, and abstract vs. pixel-perfect. In this paper, we study a further trait of sketches that has been overlooked to date, that is, they are hierarchical in terms of the levels of detail -- a person typically sketches up to various extents of detail to depict an object. This hierarchical structure is often visually distinct. In this paper, we design a novel network that is capable of cultivating sketch-specific hierarchies and exploiting them to match sketch with photo at corresponding hierarchical levels. In particular, features from a sketch and a photo are enriched using cross-modal co-attention, coupled with hierarchical node fusion at every level to form a better embedding space to conduct retrieval. Experiments on common benchmarks show our method to outperform state-of-the-arts by a significant margin.

Motivation & Objective

• To address the underexplored hierarchical nature of freehand sketches in fine-grained sketch-based image retrieval (FG-SBIR), where sketch detail varies from coarse to fine.
• To reduce the sketch-photo domain gap by explicitly modelling hierarchical semantic structures in both modalities.
• To improve cross-modal embedding quality through cross-modal co-attention and hierarchical fusion at every level.
• To enable robust retrieval regardless of sketch detail level, by learning a hierarchy that adapts to partial or incomplete sketches.
• To establish a novel, end-to-end trainable framework that discovers sketch-specific hierarchies without requiring pre-defined structure.

Proposed method

• Uses an agglomerative merging scheme to implicitly discover sketch hierarchies, where nodes (features) are merged iteratively to form higher-level representations.
• Applies a straight-through Gumbel-Softmax operation to approximate discrete merging decisions, enabling backpropagation through the non-differentiable node selection process.
• Introduces a cross-modal co-attention module that enables mutual attention between sketch and photo features at each hierarchical level.
• Employs a gating mechanism to adaptively fuse co-attended features from both modalities, filtering out mismatched or noisy parts.
• Performs hierarchical node fusion at every level, propagating refined features across levels to form the final embedding.
• Uses a distance-based ranking in the shared embedding space for retrieval, validated as superior to scalar similarity prediction.

Experimental results

Research questions

• RQ1 Can implicit, end-to-end learnable sketch hierarchies improve performance in fine-grained sketch-based image retrieval?
• RQ2 How does cross-modal co-attention at multiple hierarchical levels enhance feature alignment between sketches and photos?
• RQ3 Does hierarchical fusion with co-attention lead to more robust retrieval under varying sketch detail levels (e.g., incomplete sketches)?
• RQ4 How does the proposed method compare to explicit hierarchy modeling or non-hierarchical baselines in terms of accuracy and robustness?
• RQ5 What is the optimal number of regions (k) for co-attention in the photo branch, and how does it affect accuracy and inference time?

Key findings

• The proposed method achieves 62.45% accuracy@1 on QMUL-ChairV2 and 67.23% on SWIRE, significantly outperforming state-of-the-art methods.
• Removing cross-modal co-attention (w/o Localised-Coattn) reduces performance by 10.6% on QMUL-ChairV2 and 4.45% on QMUL-ShoeV2, demonstrating its critical role.
• Eliminating hierarchical modelling (w/o Hierarchy) drops performance by 7.21% on QMUL-ChairV2 and 3.06% on QMUL-ShoeV2, validating the importance of hierarchy.
• The model maintains strong robustness on incomplete sketches: acc.@10 remains at 87.58% (QMUL-ChairV2) and 85.64% (QMUL-ShoeV2) when 50% of strokes are dropped.
• Explicit hierarchy modeling (71.54% on SWIRE) serves as an upper bound, showing that the implicit hierarchy discovery module achieves 94.5% of the upper-bound performance.
• The optimal number of regions (k) for co-attention is 16, balancing accuracy and inference time, as increasing k raises time cost while decreasing it harms accuracy.