[Paper Review] Semantic Adversarial Network for Zero-Shot Sketch-Based Image Retrieval

Xinxun Xu, Hao Wang+2 more|arXiv (Cornell University)|May 7, 2019

Advanced Image and Video Retrieval Techniques15 references4 citations

TL;DR

This paper proposes a semantic adversarial network for zero-shot sketch-based image retrieval that enhances feature consistency with category-level word vectors and preserves intra-class discriminability via triplet loss. The end-to-end model achieves state-of-the-art performance, outperforming prior methods by over 12% on Sketchy and ~3% on TU-Berlin.

ABSTRACT

Zero-shot sketch-based image retrieval (ZS-SBIR) is a specific cross-modal retrieval task for retrieving natural images with free-hand sketches under zero-shot scenario. Previous works mostly focus on modeling the correspondence between images and sketches or synthesizing image features with sketch features. However, both of them ignore the large intra-class variance of sketches, thus resulting in unsatisfactory retrieval performance. In this paper, we propose a novel end-to-end semantic adversarial approach for ZS-SBIR. Specifically, we devise a semantic adversarial module to maximize the consistency between learned semantic features and category-level word vectors. Moreover, to preserve the discriminability of synthesized features within each training category, a triplet loss is employed for the generative module. Additionally, the proposed model is trained in an end-to-end strategy to exploit better semantic features suitable for ZS-SBIR. Extensive experiments conducted on two large-scale popular datasets demonstrate that our proposed approach remarkably outperforms state-of-the-art approaches by more than 12\% on Sketchy dataset and about 3\% on TU-Berlin dataset in the retrieval.

Motivation & Objective

• Address the large intra-class variance in free-hand sketches that limits retrieval performance in zero-shot sketch-based image retrieval.
• Improve semantic alignment between sketch and image features by leveraging category-level word vectors.
• Preserve discriminative power of synthesized features within each training category during generation.
• Develop an end-to-end trainable framework that jointly optimizes semantic consistency and feature discriminability.
• Achieve superior zero-shot retrieval performance on large-scale benchmark datasets.

Proposed method

• Introduce a semantic adversarial module that maximizes consistency between learned semantic features and category-level word vectors.
• Employ a triplet loss within the generative module to maintain discriminability of synthesized features across the same training category.
• Train the entire model end-to-end to jointly optimize semantic alignment and feature discrimination.
• Use deep neural networks to map sketches and images into a shared semantic space where semantic consistency is enforced.
• Leverage word vectors (e.g., Word2Vec or GloVe) as supervision to align features with semantic categories.
• Combine adversarial training with triplet loss to balance semantic generalization and intra-class compactness.

Experimental results

Research questions

• RQ1 Can semantic adversarial training improve alignment between sketch features and semantic category representations?
• RQ2 Does incorporating triplet loss in the generative module enhance intra-class compactness of synthesized features?
• RQ3 Can end-to-end training of the full model lead to better zero-shot retrieval performance than stage-wise or separate training?
• RQ4 How does the model perform under large intra-class variance in sketch data compared to existing methods?
• RQ5 To what extent does the proposed method outperform state-of-the-art approaches on standard ZS-SBIR benchmarks?

Key findings

• The proposed method achieves a significant improvement of over 12% on the Sketchy dataset compared to state-of-the-art approaches.
• On the TU-Berlin dataset, the model improves retrieval performance by approximately 3% over existing methods.
• The semantic adversarial module effectively enhances feature consistency with category-level word vectors, improving cross-modal alignment.
• The triplet loss component successfully preserves discriminative properties within each training category during feature synthesis.
• End-to-end training enables better optimization of both semantic consistency and feature discriminability, leading to superior retrieval accuracy.
• The model demonstrates strong generalization under zero-shot conditions, even with high intra-class variation in sketch data.