[Paper Review] Generating Embroidery Patterns Using Image-to-Image Translation

Mohammad Akif Beg, Jia Yuan Yu|arXiv (Cornell University)|Mar 5, 2020

Generative Adversarial Networks and Image Synthesis18 references4 citations

TL;DR

This paper proposes two deep learning-based methods—split style transfer and EmbGAN—for generating realistic, real-time previews of embroidered versions from user-uploaded 2D images. By modifying neural style transfer and cycle-consistent GANs, the approach produces visually accurate embroidery simulations without requiring paired training data or knowledge of embroidery-specific attributes, with EmbGAN outperforming baseline models in perceptual evaluations by 91.5% on turker preference tests.

ABSTRACT

In many scenarios in computer vision, machine learning, and computer graphics, there is a requirement to learn the mapping from an image of one domain to an image of another domain, called Image-to-image translation. For example, style transfer, object transfiguration, visually altering the appearance of weather conditions in an image, changing the appearance of a day image into a night image or vice versa, photo enhancement, to name a few. In this paper, we propose two machine learning techniques to solve the embroidery image-to-image translation. Our goal is to generate a preview image which looks similar to an embroidered image, from a user-uploaded image. Our techniques are modifications of two existing techniques, neural style transfer, and cycle-consistent generative-adversarial network. Neural style transfer renders the semantic content of an image from one domain in the style of a different image in another domain, whereas a cycle-consistent generative adversarial network learns the mapping from an input image to output image without any paired training data, and also learn a loss function to train this mapping. Furthermore, the techniques we propose are independent of any embroidery attributes, such as elevation of the image, light-source, start, and endpoints of a stitch, type of stitch used, fabric type, etc. Given the user image, our techniques can generate a preview image which looks similar to an embroidered image. We train and test our propose techniques on an embroidery dataset which consist of simple 2D images. To do so, we prepare an unpaired embroidery dataset with more than 8000 user-uploaded images along with embroidered images. Empirical results show that these techniques successfully generate an approximate preview of an embroidered version of a user image, which can help users in decision making.

Motivation & Objective

• To address the lack of automated, real-time preview systems for custom embroidery in fashion customization.
• To generate visually realistic embroidered versions of user-uploaded 2D images without relying on paired training data or embroidery-specific metadata.
• To improve customer decision-making in apparel customization by providing accurate preview simulations.
• To evaluate and compare the perceptual quality of generated embroidery-like images against ground truth digitized versions.
• To develop a method independent of stitch type, fabric, lighting, or elevation, ensuring broad applicability.

Proposed method

• Adapted neural style transfer by splitting input images into color-based sub-images and applying distinct embroidery styles per region to enhance realism.
• Modified the CycleGAN framework into EmbGAN to learn unpaired image-to-image translation between natural images and embroidered outputs using cycle-consistency loss.
• Trained models on an unpaired dataset of over 8,000 user-uploaded images and corresponding manually digitized embroidered versions.
• Used perceptual evaluation via Amazon Mechanical Turk to compare generated images against ground truth and baseline models.
• Employed a conditional adversarial framework with identity loss and cycle consistency to preserve structural and semantic content during translation.
• Integrated multiple embroidery style images per color region in split style transfer to simulate stitch variation and texture.

Experimental results

Research questions

• RQ1 Can a modified neural style transfer method produce more realistic embroidery-style previews than standard style transfer?
• RQ2 Can an unpaired image-to-image translation model like CycleGAN be effectively adapted to generate high-fidelity embroidery previews?
• RQ3 How does the performance of the proposed EmbGAN model compare to both standard CycleGAN and split style transfer in terms of visual realism?
• RQ4 To what extent do human evaluators prefer the generated embroidery previews over baseline models?
• RQ5 Does separating the image by color and applying region-specific style transfer improve the perceived authenticity of the final embroidered output?

Key findings

• Split style transfer outperformed standard neural style transfer, with 67.5% of turkers marking split style transfer results as more realistic than neural style transfer.
• EmbGAN achieved a 91.5% preference rate among turkers when compared to CycleGAN, indicating a significant improvement in visual realism.
• The generated images from EmbGAN were consistently rated as more authentic and closer to the ground truth digitized embroidery versions than those from CycleGAN.
• Perceptual evaluation confirmed that standard neural style transfer produced results that felt less realistic and less representative of actual embroidery patterns.
• The unpaired training setup with cycle-consistent loss enabled effective translation without requiring paired embroidery data, enhancing scalability.
• The method is robust to variations in image content and does not require input about stitch type, fabric, or lighting conditions.