[Paper Review] Learning Tuple Compatibility for Conditional OutfitRecommendation

Xuewen Yang, Dongliang Xie+4 more|arXiv (Cornell University)|Aug 18, 2020

Generative Adversarial Networks and Image Synthesis23 references4 citations

TL;DR

This paper proposes a Mixed Category Attention Network (MCAN) for conditional outfit recommendation that jointly models fine-grained and coarse category information to learn tuple-level compatibility among fashion items. By leveraging tuple triplet learning with hard negative sampling, MCAN enables controllable, diverse, and order-agnostic recommendations, outperforming state-of-the-art methods on both Polyvore and the new IQON dataset in compatibility prediction and 'fill-in-the-blank' tasks.

ABSTRACT

Outfit recommendation requires the answers of some challenging outfit compatibility questions such as 'Which pair of boots and school bag go well with my jeans and sweater?'. It is more complicated than conventional similarity search, and needs to consider not only visual aesthetics but also the intrinsic fine-grained and multi-category nature of fashion items. Some existing approaches solve the problem through sequential models or learning pair-wise distances between items. However, most of them only consider coarse category information in defining fashion compatibility while neglecting the fine-grained category information often desired in practical applications. To better define the fashion compatibility and more flexibly meet different needs, we propose a novel problem of learning compatibility among multiple tuples (each consisting of an item and category pair), and recommending fashion items following the category choices from customers. Our contributions include: 1) Designing a Mixed Category Attention Net (MCAN) which integrates both fine-grained and coarse category information into recommendation and learns the compatibility among fashion tuples. MCAN can explicitly and effectively generate diverse and controllable recommendations based on need. 2) Contributing a new dataset IQON, which follows eastern culture and can be used to test the generalization of recommendation systems. Our extensive experiments on a reference dataset Polyvore and our dataset IQON demonstrate that our method significantly outperforms state-of-the-art recommendation methods.

Motivation & Objective

• To address the limitation of existing outfit recommendation systems that rely only on coarse or fine category information, leading to inflexible and less accurate recommendations.
• To model fashion compatibility not just at the item level but at the tuple level (item-category pair), enabling more precise and controllable recommendations.
• To develop a recommendation system that is robust to item ordering and can handle both coarse and fine category queries flexibly.
• To contribute a new culturally diverse dataset, IQON, to support evaluation and generalization testing of outfit recommendation systems.
• To improve compatibility learning by integrating both visual aesthetics and multi-level category information through a unified attention-based architecture.

Proposed method

• Proposes a Mixed Category Attention Network (MCAN) that encodes both fine-grained and coarse category information into item representations using separate embedding layers.
• Introduces tuple triplet learning to model compatibility among item-category tuples, where positive tuples are from the same outfit and negative tuples are sampled from different outfits.
• Employs semi-hard and hard negative sampling to focus on difficult negative examples, improving the discriminative power of the compatibility space.
• Incorporates a category prediction layer (CPL) to align item sequences with category sequences, enabling order-invariant recommendations.
• Uses multi-head self-attention and cross-attention mechanisms to model interactions between items and categories, capturing complex compatibility patterns.
• Supports six distinct recommendation tasks by conditioning on partial input tuples of either coarse or fine categories, enabling diverse and controllable outfit generation.

Experimental results

Research questions

• RQ1 Can a unified model effectively learn compatibility across fashion items while incorporating both fine-grained and coarse category information?
• RQ2 How does the integration of mixed category attention improve recommendation controllability and diversity compared to sequential or pair-wise models?
• RQ3 To what extent does the proposed MCAN model generalize across different data distributions, especially with culturally distinct datasets?
• RQ4 Does the inclusion of a category prediction layer (CPL) significantly improve robustness to item ordering in outfit recommendations?
• RQ5 How effective is the proposed tuple triplet learning with hard negative sampling in improving compatibility prediction performance?

Key findings

• MCAN significantly outperforms state-of-the-art methods on both the Polyvore and IQON datasets in compatibility prediction and 'fill-in-the-blank' (FITB) tasks.
• The model achieves superior performance regardless of the order of items in an outfit, demonstrating its robustness to permutation, unlike sequential models such as BiLSTM.
• The inclusion of the category prediction layer (CPL) enables MCAN to be order-free, maintaining high performance even when test item order differs from training order.
• MCAN generates diverse and controllable recommendations across six different recommendation scenarios, including partial input of coarse or fine categories.
• The model successfully recommends visually and stylistically compatible alternatives to a target item, outperforming random selection of same-category items in visual coherence and style alignment.
• The new IQON dataset, collected under Eastern cultural context, enables better evaluation of generalization and cultural diversity in outfit recommendation systems.