[Paper Review] FashionSearchNet-v2: Learning Attribute Representations with Localization for Image Retrieval with Attribute Manipulation

Kenan E. Ak, Joo‐Hwee Lim+3 more|arXiv (Cornell University)|Nov 28, 2021

Advanced Image and Video Retrieval Techniques4 citations

TL;DR

FashionSearchNet-v2 proposes a weakly supervised, localization-aware deep learning framework for image retrieval with attribute manipulation, using class activation mapping (CAM) to isolate attribute-specific regions and triplet loss on localized regions to improve similarity learning. It outperforms state-of-the-art methods on multiple fashion and attribute-rich datasets by enabling precise, explainable attribute manipulation while maintaining other image attributes.

ABSTRACT

The focus of this paper is on the problem of image retrieval with attribute manipulation. Our proposed work is able to manipulate the desired attributes of the query image while maintaining its other attributes. For example, the collar attribute of the query image can be changed from round to v-neck to retrieve similar images from a large dataset. A key challenge in e-commerce is that images have multiple attributes where users would like to manipulate and it is important to estimate discriminative feature representations for each of these attributes. The proposed FashionSearchNet-v2 architecture is able to learn attribute specific representations by leveraging on its weakly-supervised localization module, which ignores the unrelated features of attributes in the feature space, thus improving the similarity learning. The network is jointly trained with the combination of attribute classification and triplet ranking loss to estimate local representations. These local representations are then merged into a single global representation based on the instructed attribute manipulation where desired images can be retrieved with a distance metric. The proposed method also provides explainability for its retrieval process to help provide additional information on the attention of the network. Experiments performed on several datasets that are rich in terms of the number of attributes show that FashionSearchNet-v2 outperforms the other state-of-the-art attribute manipulation techniques. Different than our earlier work (FashionSearchNet), we propose several improvements in the learning procedure and show that the proposed FashionSearchNet-v2 can be generalized to different domains other than fashion.

Motivation & Objective

• Address the challenge of image retrieval with attribute manipulation in e-commerce, where users want to modify specific attributes (e.g., collar style, color) while preserving others.
• Improve feature representation learning for multiple, spatially distinct attributes in fashion images by focusing on relevant image regions through weakly supervised localization.
• Enable flexible and explainable retrieval by learning disentangled local representations per attribute and fusing them via an attribute memory block.
• Overcome limitations of prior methods that lack spatial attention or rely on global features, which may be contaminated by irrelevant attributes.
• Generalize the approach beyond fashion to other domains by leveraging robust, localized attribute representations.

Proposed method

• Employ class activation mapping (CAM) with global average pooling (GAP) to generate spatial attention maps (AAMs) that localize relevant regions for each attribute without bounding box annotations.
• Apply ROI pooling on AAMs to extract localized, attribute-specific feature representations for each attribute, reducing interference from unrelated image parts.
• Train the network jointly with attribute classification loss and triplet ranking loss applied independently on each localized region, increasing training triplet diversity and enabling end-to-end learning of attribute similarity.
• Introduce a learnable attribute memory block that allows dynamic replacement of the query image’s attribute representation during inference, enabling real-time attribute manipulation.
• Integrate feature fusion by combining global features with localized representations to improve robustness against localization errors.
• Use a shared backbone network to extract initial features, followed by separate branches for each attribute to enable independent representation learning.

Experimental results

Research questions

• RQ1 Can weakly supervised localization via CAM improve the discriminative quality of attribute representations in image retrieval with attribute manipulation?
• RQ2 Does applying triplet loss on localized regions (rather than global features) lead to better similarity learning and retrieval performance?
• RQ3 How effective is the proposed attribute memory block in enabling dynamic, real-time attribute manipulation during retrieval?
• RQ4 To what extent does feature fusion with global features improve robustness when localization is inaccurate?
• RQ5 Can FashionSearchNet-v2 generalize to non-fasion domains and datasets with complex, multi-attribute images?

Key findings

• FashionSearchNet-v2 outperforms the state-of-the-art AMNet and other SOTA methods on multiple benchmark datasets, including Shopping100k, DARN, iMaterialist, and CelebA.
• The model achieves higher Top-1 and Top-5 retrieval accuracy on Shopping100k due to cleaner, more uniform fashion images with fewer visual distractions.
• Performance on iMaterialist is lower than on Shopping100k due to noisy, real-world street images with complex backgrounds and viewpoint variations.
• The CelebA dataset shows the highest overall performance due to frontal, well-lit face images and fewer, simpler facial attributes compared to fashion attributes.
• The inclusion of feature fusion improves robustness slightly but does not significantly boost performance, indicating that localized representations are already highly accurate.
• The model achieves efficient inference, extracting features from 10,000 images in ~60 seconds, with ROI extraction taking only ~0.002 seconds per image, making it suitable for real-time applications.