[Paper Review] An Effective Pipeline for a Real-world Clothes Retrieval System

Yang-Ho Ji, HeeJae Jun+8 more|arXiv (Cornell University)|May 26, 2020

Generative Adversarial Networks and Image Synthesis19 references4 citations

TL;DR

This paper presents a robust, three-stage pipeline for real-world clothes retrieval using detection, metric learning-based retrieval, and post-processing techniques. By integrating weighted boxes fusion and feature concatenation, the method achieved 2nd place in the DeepFashion2 2020 challenge with an Acc@10 of 0.854168 on the test set, demonstrating strong performance on large-scale fashion data.

ABSTRACT

In this paper, we propose an effective pipeline for clothes retrieval system which has sturdiness on large-scale real-world fashion data. Our proposed method consists of three components: detection, retrieval, and post-processing. We firstly conduct a detection task for precise retrieval on target clothes, then retrieve the corresponding items with the metric learning-based model. To improve the retrieval robustness against noise and misleading bounding boxes, we apply post-processing methods such as weighted boxes fusion and feature concatenation. With the proposed methodology, we achieved 2nd place in the DeepFashion2 Clothes Retrieval 2020 challenge.

Motivation & Objective

• To develop a robust, scalable pipeline for real-world fashion image retrieval on large-scale datasets.
• To improve retrieval accuracy by addressing noise and inconsistent bounding boxes in detection outputs.
• To enhance feature representation through ensemble learning and post-processing techniques.
• To achieve state-of-the-art performance in a real-world benchmark, specifically the DeepFashion2 2020 clothes retrieval challenge.

Proposed method

• The pipeline consists of three components: object detection, metric learning-based retrieval, and post-processing for improved robustness.
• Five state-of-the-art object detectors (e.g., ATSS, Cascade Mask R-CNN) were used to generate precise clothing bounding boxes.
• A metric learning model with combination of global descriptors (CGD) was employed to learn discriminative feature embeddings for retrieval.
• Weighted Boxes Fusion (WBF) was applied to merge predictions from multiple detectors, reducing inconsistency and improving localization accuracy.
• Feature concatenation of l2-normalized embeddings from multiple deep networks was used to enrich representation and boost retrieval performance.
• Post-processing techniques such as k-reciprocal re-ranking and database-side augmentation were evaluated, with only WBF and feature concatenation proving effective.

Experimental results

Research questions

• RQ1 How can a retrieval pipeline be designed to maintain high performance on large-scale, real-world fashion datasets with noisy and varied inputs?
• RQ2 To what extent do post-processing techniques like WBF and feature concatenation improve retrieval accuracy in the presence of inconsistent detection outputs?
• RQ3 Which combination of detection models and retrieval architectures yields optimal performance in a real-world clothes retrieval system?
• RQ4 Why do certain post-processing methods like PCA whitening and query expansion fail to improve performance despite theoretical promise?

Key findings

• The proposed pipeline achieved an Acc@10 of 0.854168 on the DeepFashion2 2020 test set, securing 2nd place in the challenge.
• Weighted Boxes Fusion (WBF) with five detection models (WBF₅) improved detection performance, with Acc@10 increasing from 0.869435 to 0.877192 when combined with ensemble retrieval.
• Feature concatenation of embeddings from 17 models using WBF₅ boosted Acc@10 to 0.888341 on the validation set, showing significant performance gains.
• k-reciprocal re-ranking improved Acc@10 by 0.02 when applied to 126k boxes, but did not outperform simple box reduction to 440k via WBF.
• PCA whitening and query expansion (QE) degraded performance, indicating that full-dimensional feature concatenation is preferable for this task.
• The method generalized well beyond fashion, with potential application to visual search in furniture, beauty products, and toys.