[Paper Review] Searching for Apparel Products from Images in the Wild

Son N. Tran, Ming Du+3 more|arXiv (Cornell University)|Jul 4, 2019

Video Surveillance and Tracking Methods22 references4 citations

TL;DR

This paper proposes a deep learning system for street-to-shop apparel search that localizes clothing regions in unconstrained fashion images using YOLOv3 and SSD detectors, classifies them into fine-grained categories with a multi-task CNN, and retrieves visually similar items from an online catalog using learned feature embeddings. The method achieves improved retrieval accuracy by incorporating color and attribute prediction, with human evaluation showing consistent gains from V1 to V3 embeddings.

ABSTRACT

In this age of social media, people often look at what others are wearing. In particular, Instagram and Twitter influencers often provide images of themselves wearing different outfits and their followers are often inspired to buy similar clothes.We propose a system to automatically find the closest visually similar clothes in the online Catalog (street-to-shop searching). The problem is challenging since the original images are taken under different pose and lighting conditions. The system initially localizes high-level descriptive regions (top, bottom, wristwear. . . ) using multiple CNN detectors such as YOLO and SSD that are trained specifically for apparel domain. It then classifies these regions into more specific regions such as t-shirts, tunic or dresses. Finally, a feature embedding learned using a multi-task function is recovered for every item and then compared with corresponding items in the online Catalog database and ranked according to distance. We validate our approach component-wise using benchmark datasets and end-to-end using human evaluation.

Motivation & Objective

• To enable accurate visual search for apparel items from real-world fashion images posted on social media.
• To address challenges in clothing detection due to unconstrained poses, lighting, occlusions, and multi-layered outfits.
• To improve retrieval accuracy by learning discriminative feature embeddings that capture product type, color, and other attributes.
• To validate the system end-to-end using human evaluation and benchmark datasets.
• To reduce reliance on separate embedding networks by leveraging features from a fine-grained classification model.

Proposed method

• Employ YOLOv3 and SSD detectors trained on apparel-specific data to localize high-level regions (e.g., top, bottom, wristwear) in fashion images.
• Use a hierarchical multi-task CNN to classify detected regions into 146 fine-grained apparel categories, with joint learning of product type, color, pattern, and sleeve length.
• Extract feature embeddings from the final layer before softmax for both query images and catalog items to enable similarity computation.
• Restrict similarity comparisons to items within the same fine-grained class to improve retrieval consistency.
• Apply background augmentation during training to make embeddings robust to background color interference.
• Use human evaluation via MTurk to compare retrieval quality between different embedding versions (V1 to V3).

Experimental results

Research questions

• RQ1 Can multi-box object detectors trained on apparel-specific data outperform general-purpose detectors like OpenImages in localizing clothing regions in unconstrained fashion images?
• RQ2 How does multi-task learning with attributes (color, pattern, sleeve length) improve the quality of learned feature embeddings for apparel retrieval?
• RQ3 To what extent does background augmentation during training improve embedding robustness to background clutter in real-world images?
• RQ4 How do the different embedding variants (V1, V2, V3) compare in end-to-end retrieval performance under human evaluation?
• RQ5 Can a fine-grained classification network serve as an effective end-to-end embedding extractor without requiring a separate metric learning stage?

Key findings

• The fine-grained classification accuracy for tops improved from 0.52 (V1) to 0.80 (V2) after data cleaning and multi-task learning, indicating significant gains from improved data and joint learning.
• The system achieved a 75% preference rate for V2 over V1 in human evaluation, demonstrating that color-aware embeddings improve retrieval quality.
• The preference for V3 over V2 reached 67%, showing consistent improvement from adding 13 additional classification tasks to the multi-task network.
• The model maintained strong performance on real-world fashion images (e.g., 0.71 accuracy for dresses), indicating good generalization beyond catalog images.
• Despite challenges with multi-layered tops due to single-bounding-box detection, the system showed robustness to background color when trained with background augmentation.
• The use of embeddings from a fine-grained classification network eliminated the need for a separate metric learning network, simplifying the pipeline while maintaining high performance.