[Paper Review] Styling with Attention to Details

Ayushi Dalmia, Sachindra Joshi+2 more|arXiv (Cornell University)|Jul 3, 2018

Recommender Systems and Techniques5 references3 citations

TL;DR

This paper proposes a sequence-to-sequence model with attention mechanisms to recommend stylish, complementary fashion items based on user-generated social media content. By leveraging likes and comments from fashion posts, the model learns style relationships and outperforms baseline methods by 28% in top-1 accuracy and 24% in mean reciprocal rank for retrieval tasks.

ABSTRACT

Fashion as characterized by its nature, is driven by style. In this paper, we propose a method that takes into account the style information to complete a given set of selected fashion items with a complementary fashion item. Complementary items are those items that can be worn along with the selected items according to the style. Addressing this problem facilitates in automatically generating stylish fashion ensembles leading to a richer shopping experience for users. Recently, there has been a surge of online social websites where fashion enthusiasts post the outfit of the day and other users can like and comment on them. These posts contain a gold-mine of information about style. In this paper, we exploit these posts to train a deep neural network which captures style in an automated manner. We pose the problem of predicting complementary fashion items as a sequence to sequence problem where the input is the selected set of fashion items and the output is a complementary fashion item based on the style information learned by the model. We use the encoder decoder architecture to solve this problem of completing the set of fashion items. We evaluate the goodness of the proposed model through a variety of experiments. We empirically observe that our proposed model outperforms competitive baseline like apriori algorithm by ~28 in terms of accuracy for top-1 recommendation to complete the fashion ensemble. We also perform retrieval based experiments to understand the ability of the model to learn style and rank the complementary fashion items and find that using attention in our encoder decoder model helps in improving the mean reciprocal rank by ~24. Qualitatively we find the complementary fashion items generated by our proposed model are richer than the apriori algorithm.

Motivation & Objective

• To address the challenge of recommending complementary fashion items that align with user style, beyond simple co-occurrence or similarity-based recommendations.
• To model the subjective and evolving nature of fashion style using data from social media platforms like Polyvore and Chictopia.
• To develop a data-driven, generalizable method that captures complex attribute-level interactions (e.g., color, pattern) in fashion ensembles.
• To improve recommendation accuracy and ranking quality by incorporating attention mechanisms in an encoder-decoder architecture.
• To move beyond static purchase history and leverage real-time, user-driven style signals from social media for dynamic style modeling.

Proposed method

• Formulates the fashion ensemble completion task as a sequence-to-sequence problem, where the input is a set of fashion items and the output is a complementary item.
• Employs an encoder-decoder architecture with a deep neural network to learn contextual representations of fashion ensembles from social media posts.
• Uses attention mechanisms in the decoder to dynamically focus on relevant features of the input fashion set during prediction.
• Trains the model on parsed textual descriptions from social media posts, using social signals (likes, comments) as implicit style indicators.
• Leverages fashion taxonomy and attribute-level features (color, pattern, type) to enrich input representations and improve generalization.
• Optimizes the model using cross-entropy loss for sequence generation, with fine-tuning based on retrieval performance metrics like mean reciprocal rank.

Experimental results

Research questions

• RQ1 Can social media signals such as likes and comments be effectively used to model subjective and evolving fashion style?
• RQ2 How well can a sequence-to-sequence model with attention learn to generate stylistically coherent complementary fashion items compared to traditional rule-based or association-based methods?
• RQ3 To what extent does attention improve the ranking quality of recommended fashion items in retrieval settings?
• RQ4 Can a deep learning model generalize across diverse fashion ensembles without relying on handcrafted rules or curated style datasets?
• RQ5 How does the model perform in comparison to baselines like the apriori algorithm in terms of top-1 accuracy and retrieval effectiveness?

Key findings

• The proposed sequence-to-sequence model with attention achieves a 28% improvement in top-1 recommendation accuracy over the apriori algorithm.
• The attention mechanism improves mean reciprocal rank (MRR) by approximately 24% in retrieval-based evaluation, indicating better ranking of relevant items.
• Qualitative analysis shows that the model generates more diverse and stylistically richer complementary fashion items than the apriori baseline.
• The model effectively captures complex style relationships, such as color and pattern compatibility, from unstructured social media text.
• The use of social signals enables the model to learn current, up-to-date style trends, avoiding the staleness of traditional purchase-based methods.
• The model generalizes well across varied fashion ensembles, demonstrating robustness in generating relevant recommendations even for rare or complex combinations.