[Paper Review] TryOnGAN: Body-Aware Try-On via Layered Interpolation

Kathleen M. Lewis, Srivatsan Varadharajan+1 more|arXiv (Cornell University)|Jan 6, 2021

Generative Adversarial Networks and Image Synthesis14 references4 citations

TL;DR

TryOnGAN proposes a novel, pose-conditioned StyleGAN2-based method for photorealistic virtual try-on that enables body-aware garment transfer via per-layer latent space interpolation, achieving state-of-the-art results on 512×512 images without paired training data. It preserves identity, skin color, and body shape while seamlessly blending garments, with superior performance on both real and generated images due to optimized, layer-specific interpolation coefficients.

ABSTRACT

Given a pair of images-target person and garment on another person-we automatically generate the target person in the given garment. Previous methods mostly focused on texture transfer via paired data training, while overlooking body shape deformations, skin color, and seamless blending of garment with the person. This work focuses on those three components, while also not requiring paired data training. We designed a pose conditioned StyleGAN2 architecture with a clothing segmentation branch that is trained on images of people wearing garments. Once trained, we propose a new layered latent space interpolation method that allows us to preserve and synthesize skin color and target body shape while transferring the garment from a different person. We demonstrate results on high resolution 512x512 images, and extensively compare to state of the art in try-on on both latent space generated and real images.

Motivation & Objective

• To address the limitations of prior virtual try-on methods that overlook body shape deformation, skin color consistency, and seamless garment blending.
• To develop a method that performs high-fidelity try-on on high-resolution (512×512) images without requiring paired training data.
• To enable accurate transfer of complex garment textures and patterns while preserving the target person’s identity and body structure.
• To design a latent space optimization strategy that automatically computes optimal interpolation coefficients per layer, avoiding manual tuning.
• To demonstrate the method's generalization across diverse body types, poses, and garment styles using unpaired data.

Proposed method

• A modified StyleGAN2 architecture is trained on 100K unpaired fashion images, incorporating a pose-conditioned generator and a clothing segmentation branch.
• The method performs per-layer latent space interpolation by optimizing coefficients for each layer of the generator to blend the target person and garment images.
• Optimization is guided by a multi-component loss function including identity loss (for face and hair), localization loss (to restrict edits to garment regions), and garment loss (to transfer shape, color, and texture).
• For real image try-on, a projection step maps the input image to the StyleGAN latent space before interpolation; for generated images, this step is bypassed.
• The architecture is designed to segment the generated image during optimization, enabling accurate region-specific editing without external segmentation models.
• The method supports cross-pose try-on by explicitly conditioning the generator on 2D human body pose, enabling control over the target person's pose during synthesis.

Experimental results

Research questions

• RQ1 Can a GAN-based method achieve high-fidelity virtual try-on without paired training data, preserving identity, skin color, and body shape?
• RQ2 How does per-layer latent space interpolation compare to greedy or fixed-parameter methods in terms of preserving garment details and identity features?
• RQ3 To what extent can the method transfer complex textures and geometric patterns from the garment image to the target person?
• RQ4 How does pose conditioning improve the quality and controllability of try-on results across different body poses?
• RQ5 What is the impact of the loss function components (identity, localization, garment) on the final synthesis quality and semantic consistency?

Key findings

• TryOnGAN achieves state-of-the-art performance on 512×512 real and generated images, outperforming prior methods in photorealism, body shape fidelity, and skin color preservation.
• The per-layer optimization approach significantly improves detail transfer—such as sleeve length and texture patterns—compared to greedy search methods, as shown in qualitative comparisons.
• On generated images, TryOnGAN successfully transfers high-frequency details like geometric patterns and complex textures, demonstrating the full potential of the optimization method.
• The ablation study confirms that all three loss components (identity, localization, garment) are essential for producing photo-realistic results with correct semantic editing.
• The method effectively synthesizes skin and body parts consistently, such as showing more arm exposure when transferring a short-sleeved shirt to a long-sleeved target.
• Despite strong results, performance degrades on extreme poses and underrepresented garments, and real image results are limited by current latent space projection quality.