[Paper Review] Bi-Directional Domain Translation for Zero-Shot Sketch-Based Image Retrieval

Jiangtong Li, Zhixin Ling+2 more|arXiv (Cornell University)|Nov 29, 2019

Advanced Image and Video Retrieval Techniques59 references4 citations

TL;DR

This paper proposes a Bi-Directional Domain Translation (BDT) framework for Zero-Shot Sketch-Based Image Retrieval (ZS-SBIR), disentangling image features into structure and appearance components to enable cross-domain translation and structure-based retrieval. The method achieves state-of-the-art performance, improving retrieval accuracy by ~8% on Sketchy and over 5% on TU-Berlin compared to prior approaches.

ABSTRACT

The goal of Sketch-Based Image Retrieval (SBIR) is using free-hand sketches to retrieve images of the same category from a natural image gallery. However, SBIR requires all categories to be seen during training, which cannot be guaranteed in real-world applications. So we investigate more challenging Zero-Shot SBIR (ZS-SBIR), in which test categories do not appear in the training stage. Traditional SBIR methods are prone to be category-based retrieval and cannot generalize well from seen categories to unseen ones. In contrast, we disentangle image features into structure features and appearance features to facilitate structure-based retrieval. To assist feature disentanglement and take full advantage of disentangled information, we propose a Bi-directional Domain Translation (BDT) framework for ZS-SBIR, in which the image domain and sketch domain can be translated to each other through disentangled structure and appearance features. Finally, we perform retrieval in both structure feature space and image feature space. Extensive experiments demonstrate that our proposed approach remarkably outperforms state-of-the-art approaches by about 8% on the Sketchy dataset and over 5% on the TU-Berlin dataset.

Motivation & Objective

• To address the challenge of zero-shot sketch-based image retrieval where test categories are unseen during training.
• To overcome the limitation of category-based retrieval in traditional SBIR methods that fail to generalize to unseen categories.
• To enable effective cross-domain retrieval by disentangling image features into structure and appearance components.
• To develop a bi-directional translation framework that leverages disentangled features to translate between sketch and image domains.
• To improve retrieval performance by leveraging both structure and image feature spaces in a unified framework.

Proposed method

• Disentangle image features into structure and appearance components using a disentanglement module.
• Design a bi-directional domain translation framework that translates between sketch and image domains using disentangled features.
• Train the model to reconstruct images and sketches from their disentangled components, enforcing consistency across domains.
• Perform retrieval in both structure feature space and image feature space to enhance generalization.
• Use adversarial training and cycle-consistency loss to improve the quality and alignment of translated features.
• Jointly optimize the disentanglement and translation components via end-to-end learning.

Experimental results

Research questions

• RQ1 Can disentangled structure and appearance features improve zero-shot generalization in sketch-based image retrieval?
• RQ2 How effective is bi-directional domain translation in aligning sketch and image domains for unseen categories?
• RQ3 Does structure-based retrieval outperform category-based retrieval in zero-shot settings?
• RQ4 To what extent does joint optimization of disentanglement and translation improve retrieval accuracy?
• RQ5 How does the proposed method compare to state-of-the-art approaches on benchmark datasets?

Key findings

• The proposed BDT framework achieves a significant improvement of approximately 8% over state-of-the-art methods on the Sketchy dataset.
• On the TU-Berlin dataset, the method improves retrieval performance by more than 5% compared to existing approaches.
• The disentanglement of structure and appearance features enables better generalization to unseen categories in zero-shot settings.
• Retrieval in both structure and image feature spaces leads to more robust and accurate results than using either space alone.
• The bi-directional translation mechanism effectively preserves semantic and structural consistency between sketches and images.
• The model demonstrates strong zero-shot generalization, effectively retrieving images for categories not seen during training.