[Paper Review] Interpretable Partitioned Embedding for Customized Fashion Outfit Composition

Zunlei Feng, Zhenyun Yu+4 more|arXiv (Cornell University)|Jun 13, 2018

Generative Adversarial Networks and Image Synthesis3 references3 citations

TL;DR

This paper proposes a partitioned embedding network that learns interpretable, attribute-specific representations for fashion items using auto-encoders, supervised attribute labels, and adversarial independence constraints. The method enables personalized outfit composition with transparent, attribute-level matching scores, significantly improving interpretability and user-desirability over existing deep learning approaches.

ABSTRACT

Intelligent fashion outfit composition becomes more and more popular in these years. Some deep learning based approaches reveal competitive composition recently. However, the unexplainable characteristic makes such deep learning based approach cannot meet the the designer, businesses and consumers' urge to comprehend the importance of different attributes in an outfit composition. To realize interpretable and customized fashion outfit compositions, we propose a partitioned embedding network to learn interpretable representations from clothing items. The overall network architecture consists of three components: an auto-encoder module, a supervised attributes module and a multi-independent module. The auto-encoder module serves to encode all useful information into the embedding. In the supervised attributes module, multiple attributes labels are adopted to ensure that different parts of the overall embedding correspond to different attributes. In the multi-independent module, adversarial operation are adopted to fulfill the mutually independent constraint. With the interpretable and partitioned embedding, we then construct an outfit composition graph and an attribute matching map. Given specified attributes description, our model can recommend a ranked list of outfit composition with interpretable matching scores. Extensive experiments demonstrate that 1) the partitioned embedding have unmingled parts which corresponding to different attributes and 2) outfits recommended by our model are more desirable in comparison with the existing methods.

Motivation & Objective

• Address the lack of interpretability in deep learning-based fashion outfit composition systems.
• Enable designers, businesses, and consumers to understand the contribution of individual attributes (e.g., color, shape) in outfit decisions.
• Develop a weakly-supervised framework that learns from raw outfit data without requiring quality scores.
• Support dynamic adaptation to evolving fashion trends through iterative model updates.
• Provide personalized outfit recommendations based on user-specified attribute preferences with interpretable matching scores.

Proposed method

• Employ an auto-encoder module to compress visual features of clothing items into a shared embedding space.
• Introduce a supervised attributes module that enforces specific embedding sub-vectors to correspond to predefined attributes (e.g., color, shape).
• Apply a multi-independent module using adversarial training to ensure each embedding part is mutually independent and only captures its designated attribute.
• Construct a fashion composition graph using partitioned embeddings, where nodes represent attribute clusters and edges represent item compatibility.
• Build an attribute matching map that quantifies the contribution of each attribute to the overall outfit composition score.
• Enable iterative model updates by reweighting the composition graph using new outfit data, allowing real-time adaptation to emerging trends.

Experimental results

Research questions

• RQ1 Can we learn a partitioned embedding that disentangles different fashion attributes (e.g., color, shape) into distinct, interpretable components?
• RQ2 How effective is the proposed multi-independent constraint in ensuring that each embedding part captures only one attribute?
• RQ3 Can the model generate personalized outfit recommendations based on user-specified attributes with transparent, attribute-level matching scores?
• RQ4 How well does the model adapt to evolving fashion trends when new data is incrementally added?
• RQ5 Does the weakly-supervised approach, relying only on outfit-level data without quality scores, outperform existing methods in user-desirability and interpretability?

Key findings

• The partitioned embedding successfully isolates different attributes into distinct sub-vectors, with color and shape components showing clear clustering in t-SNE visualization.
• The multi-independent constraint effectively prevents cross-attribute leakage, as evidenced by bags with similar textures or styles being grouped together despite differing colors and shapes.
• Outfits recommended by the model received higher user-desirability scores compared to baseline methods, particularly when users specified attributes like color or shape.
• The attribute matching scores provided by the model align with human perception—color and shape attributes contributed more to the final composition score than residual attributes.
• The iterative model update mechanism successfully tracked fashion trends over time, with visualizations showing consistent alignment with annual style shifts from 2007 to 2017.
• The model generalizes well to new data, maintaining high performance even when updating the composition graph with new outfit datasets without retraining from scratch.