[Paper Review] Theme-Matters: Fashion Compatibility Learning via Theme Attention

Jui-Hsin Lai, Bo Wu+4 more|arXiv (Cornell University)|Dec 12, 2019

Generative Adversarial Networks and Image Synthesis26 references6 citations

TL;DR

This paper proposes a theme-attention model for fashion compatibility learning, introducing the first real-world dataset (Fashion32) with 14K outfits and 32 theme labels. By leveraging category-specific embedding spaces and theme-aware attention mechanisms, the method outperforms SOTA baselines in theme-conditioned compatibility prediction, achieving 94.26% AUC on outfit-level compatibility and 76.87% accuracy in fitting-the-bag tasks.

ABSTRACT

Fashion compatibility learning is important to many fashion markets such as outfit composition and online fashion recommendation. Unlike previous work, we argue that fashion compatibility is not only a visual appearance compatible problem but also a theme-matters problem. An outfit, which consists of a set of fashion items (e.g., shirt, suit, shoes, etc.), is considered to be compatible for a "dating" event, yet maybe not for a "business" occasion. In this paper, we aim at solving the fashion compatibility problem given specific themes. To this end, we built the first real-world theme-aware fashion dataset comprising 14K around outfits labeled with 32 themes. In this dataset, there are more than 40K fashion items labeled with 152 fine-grained categories. We also propose an attention model learning fashion compatibility given a specific theme. It starts with a category-specific subspace learning, which projects compatible outfit items in certain categories to be close in the subspace. Thanks to strong connections between fashion themes and categories, we then build a theme-attention model over the category-specific embedding space. This model associates themes with the pairwise compatibility with attention, and thus compute the outfit-wise compatibility. To the best of our knowledge, this is the first attempt to estimate outfit compatibility conditional on a theme. We conduct extensive qualitative and quantitative experiments on our new dataset. Our method outperforms the state-of-the-art approaches.

Motivation & Objective

• To address the limitation of existing fashion compatibility models that ignore thematic context, such as 'business' or 'casual' occasions.
• To develop a method that estimates outfit compatibility conditional on specific themes, rather than relying solely on visual appearance.
• To create a high-quality, real-world dataset with fine-grained fashion categories and theme annotations to support theme-aware fashion modeling.
• To enable more accurate and context-aware fashion recommendation by explicitly modeling the relationship between fashion themes and item compatibility.

Proposed method

• The method employs category-specific subspace learning via triplet networks and embedding masks to project fashion items into discriminative subspaces per category.
• It constructs a theme-attention mechanism that dynamically weights pairwise compatibilities based on the input theme, enabling theme-conditioned compatibility estimation.
• The model uses a multi-head attention mechanism over category-specific embeddings to compute theme-aware compatibility scores for all item pairs in an outfit.
• The final outfit compatibility score is aggregated from attention-weighted pairwise compatibilities, enabling end-to-end training with supervision on outfit-level labels.
• The framework is trained using contrastive loss on triplet pairs and optimized via backpropagation with end-to-end learning.
• The model is evaluated on a newly curated dataset, Fashion32, which includes 14K outfits, 32 themes, and 40K items annotated with 152 fine-grained categories.

Experimental results

Research questions

• RQ1 Can fashion compatibility be significantly improved by conditioning on specific themes such as 'business' or 'casual'?
• RQ2 How does incorporating fine-grained fashion categories enhance theme-aware compatibility modeling compared to coarse categories?
• RQ3 To what extent does a theme-attention mechanism improve compatibility prediction over baseline models that ignore thematic context?
• RQ4 How does the proposed method generalize across diverse themes like 'sports', 'office', 'Japanese', and 'nature'?
• RQ5 Does the use of category-specific embedding spaces lead to better compatibility estimation than a shared embedding space?

Key findings

• The proposed model achieves 94.26% AUC on the total test set for outfit compatibility prediction, outperforming all baselines in the theme-aware setting.
• On the 'business' theme, the model reaches 94.75% AUC and 78.18% FITB accuracy, significantly outperforming the next best method.
• For the 'school' theme, the model achieves 96.92% AUC and 82.52% FITB accuracy, demonstrating strong generalization across diverse occasions.
• In the 'thin' and 'tall' body-type theme groups, the model achieves 94.63% and 93.94% AUC respectively, showing robustness to body shape variations.
• The model consistently outperforms the Type-Aware and Theme-Graph baselines across all theme groups, including 'office', 'sports', and 'female/male' gender themes.
• The ablation study confirms that both category-specific embeddings and theme-attention are essential, with performance dropping by over 5% when either component is removed.