[Paper Review] 3D Virtual Garment Modeling from RGB Images

Yi Xu, Shanglin Yang+4 more|arXiv (Cornell University)|Jul 31, 2019

3D Shape Modeling and Analysis40 references4 citations

TL;DR

This paper presents a novel method for generating 3D textured virtual garment models from just two RGB images—front and back—without requiring specialized hardware or human models. It uses a multi-task deep learning network (JFNet) to predict fashion landmarks and semantic part segmentation, which guide template mesh deformation and texture mapping, enabling accurate, scalable 3D garment modeling from e-commerce photos.

ABSTRACT

We present a novel approach that constructs 3D virtual garment models from photos. Unlike previous methods that require photos of a garment on a human model or a mannequin, our approach can work with various states of the garment: on a model, on a mannequin, or on a flat surface. To construct a complete 3D virtual model, our approach only requires two images as input, one front view and one back view. We first apply a multi-task learning network called JFNet that jointly predicts fashion landmarks and parses a garment image into semantic parts. The predicted landmarks are used for estimating sizing information of the garment. Then, a template garment mesh is deformed based on the sizing information to generate the final 3D model. The semantic parts are utilized for extracting color textures from input images. The results of our approach can be used in various Virtual Reality and Mixed Reality applications.

Motivation & Objective

• To enable 3D virtual garment modeling from readily available product photos, such as those from e-commerce sites, without requiring specialized capture setups or human models.
• To address the high cost and complexity of traditional 3D garment modeling in fashion and VR/AR applications.
• To develop a method that works across diverse garment states—on models, mannequins, or flat surfaces—using only two input images.
• To support scalable, low-cost 3D garment modeling for applications like virtual try-on and digital fashion by leveraging semantic parsing and size estimation.

Proposed method

• A multi-task learning network, JFNet, is trained to jointly predict fashion landmarks and perform semantic part segmentation on garment images.
• Landmark predictions are used to estimate key garment measurements (e.g., sleeve length, chest width), which guide the deformation of a 3D template mesh.
• Semantic part segmentation enables precise texture extraction from input images by mapping 2D image parts onto a 2D reference texture for 3D model mapping.
• The 3D template mesh is deformed based on estimated sizing information to match the proportions of the input garment.
• Textures are lifted from segmented parts and projected onto the deformed 3D mesh to produce a fully textured 3D garment model.
• The system supports single-image input under the assumption of front-back symmetry, expanding usability with minimal input.

Experimental results

Research questions

• RQ1 Can 3D virtual garments be accurately reconstructed from only two RGB images of a garment in various states (on model, mannequin, or flat)?
• RQ2 Can a single deep learning model effectively predict both fashion landmarks and semantic part segmentation for garment images?
• RQ3 How accurately can sizing information be estimated from 2D image landmarks to deform a 3D template mesh into a plausible 3D garment?
• RQ4 Can texture extraction from segmented parts produce visually plausible and consistent results on the final 3D model?
• RQ5 To what extent can the method generalize across different garment types (e.g., T-shirts, pants) and styles (e.g., fit, length) without requiring retraining?

Key findings

• The proposed JFNet achieved a mean Intersection over Union (mIOU) of 0.725 for part segmentation on tops and 0.968 on pants, outperforming baseline methods like DeepLabV3+.
• The method achieved a mean error (NE) of 0.031 for landmark prediction on tops and 0.022 on pants, demonstrating high accuracy in keypoint localization.
• 3D modeling results show visually plausible garments that reflect input image characteristics, such as sleeve width, length, and fit, indicating effective shape transfer.
• The system successfully generates complete 3D textured models from two images, even when garments are not worn on a human, including flat or mannequin states.
• The approach enables texture mapping with minimal distortion, though some local deformation occurs when source and target contours differ significantly.
• The method supports single-image input under symmetry assumptions, increasing flexibility for real-world e-commerce photo sets.