[Paper Review] Fashioning with Networks: Neural Style Transfer to Design Clothes

Prutha Date, Ashwinkumar Ganesan+1 more|arXiv (Cornell University)|Jul 31, 2017

Generative Adversarial Networks and Image Synthesis20 references21 citations

TL;DR

This paper proposes a neural style transfer pipeline that personalizes clothing design by learning a user's fashion preferences from their closet images and generating custom apparel using content and style loss optimization. The method successfully transfers multiple style patterns onto a user-defined garment outline, producing personalized, visually coherent designs that reflect individual taste.

ABSTRACT

Convolutional Neural Networks have been highly successful in performing a host of computer vision tasks such as object recognition, object detection, image segmentation and texture synthesis. In 2015, Gatys et. al [7] show how the style of a painter can be extracted from an image of the painting and applied to another normal photograph, thus recreating the photo in the style of the painter. The method has been successfully applied to a wide range of images and has since spawned multiple applications and mobile apps. In this paper, the neural style transfer algorithm is applied to fashion so as to synthesize new custom clothes. We construct an approach to personalize and generate new custom clothes based on a users preference and by learning the users fashion choices from a limited set of clothes from their closet. The approach is evaluated by analyzing the generated images of clothes and how well they align with the users fashion style.

Motivation & Objective

• To develop a personalized fashion design system that generates new clothing patterns based on a user’s unique style preferences.
• To address the challenge of fashion personalization beyond retrieval or recommendation by synthesizing novel, user-specific apparel designs.
• To evaluate whether neural style transfer can effectively transfer complex fashion styles while preserving garment structure and user preferences.
• To explore the impact of multiple style inputs on the quality and coherence of generated clothing designs.

Proposed method

• The system first learns user preferences by extracting features from a limited set of closet images using a pre-trained 19-layer VGG-Net.
• A user-provided garment outline (UCO) serves as the content image, while style features are extracted from multiple user-provided clothing images.
• The method optimizes a generated image using a total loss function combining content loss (L2 distance between feature maps of UCO and generated image) and style loss (Gram matrix distance between style images and generated image).
• The loss is minimized using LBFGS optimization with adaptive weights α and β for content and style, respectively.
• Post-processing applies a mask from the UCO to isolate the garment and remove background, producing the final design.
• The system evaluates style transfer success by training an SVM on VGG-16 features to classify attribute combinations in generated images.

Experimental results

Research questions

• RQ1 Can neural style transfer effectively personalize clothing design using only a small set of user-provided closet images?
• RQ2 How does the use of multiple style images affect the coherence and recognizability of the generated garment design?
• RQ3 To what extent can a classifier detect the intended style attributes in generated clothing images, indicating successful style transfer?
• RQ4 Does increasing the number of style inputs degrade the quality of the generated design, and if so, why?

Key findings

• The F1-score for attribute classification on generated images remained consistently above baseline, indicating successful transfer of style attributes from user-provided images.
• When generating images using styles from the same training data, the F1-score was higher than when using separate test images, suggesting better generalization on familiar style patterns.
• Increasing the number of style images used for generation led to a drop in F1-score, indicating that too many style functions degrade result quality and coherence.
• Multiple distinct style images (e.g., different knits) applied to the same content image produced visually distinct and plausible blended designs, demonstrating effective style fusion.
• The system successfully generated new, personalized clothing designs that reflect the user’s fashion sense, as validated through both quantitative and qualitative evaluation.