[Paper Review] Snap and Find: Deep Discrete Cross-domain Garment Image Retrieval

Yadan Luo, Ziwei Wang+3 more|arXiv (Cornell University)|Apr 5, 2019

Advanced Image and Video Retrieval Techniques31 references4 citations

TL;DR

This paper proposes Deep Multi-task Cross-domain Hashing (DMCH), a joint learning framework that models sequential clothing attributes and learns attention-aware visual features for cross-domain garment image retrieval. By leveraging sequential attribute learning to guide spatial attention without requiring landmark annotations, DMCH achieves 306× faster inference than state-of-the-art models while outperforming them on both retrieval accuracy and attribute prediction on DeepFashion and DARN datasets.

ABSTRACT

With the increasing number of online stores, there is a pressing need for intelligent search systems to understand the item photos snapped by customers and search against large-scale product databases to find their desired items. However, it is challenging for conventional retrieval systems to match up the item photos captured by customers and the ones officially released by stores, especially for garment images. To bridge the customer- and store- provided garment photos, existing studies have been widely exploiting the clothing attributes ( extit{e.g.,} black) and landmarks ( extit{e.g.,} collar) to learn a common embedding space for garment representations. Unfortunately they omit the sequential correlation of attributes and consume large quantity of human labors to label the landmarks. In this paper, we propose a deep multi-task cross-domain hashing termed extit{DMCH}, in which cross-domain embedding and sequential attribute learning are modeled simultaneously. Sequential attribute learning not only provides the semantic guidance for embedding, but also generates rich attention on discriminative local details ( extit{e.g.,} black buttons) of clothing items without requiring extra landmark labels. This leads to promising performance and 306$ imes$ boost on efficiency when compared with the state-of-the-art models, which is demonstrated through rigorous experiments on two public fashion datasets.

Motivation & Objective

• To address the challenge of matching user-captured garment images with professionally shot product images in cross-domain retrieval.
• To reduce reliance on costly landmark annotations by modeling sequential clothing attributes as semantic guidance for visual attention.
• To improve retrieval efficiency through compact 128-bit binary codes while maintaining high accuracy.
• To jointly optimize cross-domain image retrieval and sequential attribute prediction in an end-to-end deep learning framework.
• To enhance fine-grained discrimination by capturing semantic correlations among clothing attributes in a linguistically ordered sequence.

Proposed method

• Proposes a deep multi-task learning framework that jointly trains for cross-domain image retrieval and sequential attribute prediction.
• Uses a sequence of ordered attributes (e.g., 'black', 'silk', 'short-sleeve') as semantic supervision to guide spatial attention in convolutional features.
• Applies a triplet loss to learn discriminative binary embeddings that preserve similarity between user and shop domain images of the same garment.
• Employs a Siamese-like network architecture with shared backbones to extract visual features from user and shop images.
• Introduces an attention module that highlights discriminative local regions (e.g., buttons, collars) based on predicted attribute sequences.
• Fixes the binary code length at 128 bits to enable efficient Hamming space search, significantly improving query speed.

Experimental results

Research questions

• RQ1 Can modeling sequential clothing attributes improve cross-domain garment image retrieval performance without requiring manual landmark annotations?
• RQ2 How does sequential attribute learning enhance attention to fine-grained visual details in garment images?
• RQ3 To what extent can joint learning of attribute prediction and image retrieval improve retrieval accuracy and efficiency?
• RQ4 How does the proposed method compare to state-of-the-art models in terms of both retrieval performance and computational efficiency?
• RQ5 What is the impact of linguistic ordering in attribute sequences on the model's ability to generate meaningful and accurate descriptions?

Key findings

• DMCH achieves a ROUGE-L score of 51.0 on the DARN dataset and 44.1 on DeepFashion, significantly outperforming prior methods like DARN (32.5) and FashionNet (35.8).
• On the DARN dataset, DMCH achieves a CIDEr score of 98.6, far exceeding FashionNet (76.5) and DARN (53.2), demonstrating superior sequential attribute generation.
• The model improves retrieval accuracy by leveraging sequential attribute learning to guide spatial attention, reducing reliance on annotated landmarks.
• DMCH achieves a 306× speedup in inference efficiency compared to state-of-the-art models, due to compact 128-bit binary codes.
• Attention visualization confirms that the model correctly identifies discriminative regions such as sleeves and collars, and preserves semantic relationships between attributes like 'lace' and 'lacing'.
• t-SNE visualization of the embedding space shows that similar garments are clustered together, even when images vary in pose, lighting, or background.