[Paper Review] VOGUE: Try-On by StyleGAN Interpolation Optimization

Kathleen M. Lewis, Srivatsan Varadharajan+1 more|arXiv (Cornell University)|Jan 6, 2021

Generative Adversarial Networks and Image Synthesis20 citations

TL;DR

VOGUE proposes a pose-conditioned StyleGAN2 latent space interpolation method to generate photo-realistic, high-resolution (512×512) virtual try-on results by seamlessly blending body shape, skin tone, and hair from a target person with garment details—such as folds, material, and pattern—from a reference garment image. The key contribution is automatic optimization of interpolation coefficients per latent layer to achieve realistic, deformation-aware garment transfer.

ABSTRACT

Given an image of a target person and an image of another person wearing a garment, we automatically generate the target person in the given garment. At the core of our method is a pose-conditioned StyleGAN2 latent space interpolation, which seamlessly combines the areas of interest from each image, i.e., body shape, hair, and skin color are derived from the target person, while the garment with its folds, material properties, and shape comes from the garment image. By automatically optimizing for interpolation coefficients per layer in the latent space, we can perform a seamless, yet true to source, merging of the garment and target person. Our algorithm allows for garments to deform according to the given body shape, while preserving pattern and material details. Experiments demonstrate state-of-the-art photo-realistic results at high resolution ($512 imes 512$).

Motivation & Objective

• To address the challenge of generating realistic, high-resolution virtual try-on images that preserve both the target person's identity and the garment's material and pattern details.
• To enable natural garment deformation according to the target person's body shape without manual tuning.
• To achieve seamless fusion of source and target image attributes—such as skin color, hair, and body shape—using latent space interpolation.
• To develop an automatic optimization strategy for interpolation coefficients across StyleGAN2's latent layers to improve realism and consistency.

Proposed method

• The method uses a pose-conditioned StyleGAN2 generator to condition latent space interpolation on the target person's pose.
• It performs latent space interpolation between the target person and garment image, with coefficients optimized per layer to balance identity and garment fidelity.
• Optimization is performed end-to-end using a differentiable loss that preserves identity, texture, and structural details.
• The approach leverages the disentangled latent space of StyleGAN2 to isolate and transfer specific attributes—e.g., body shape from the target, garment from the reference.
• The method ensures that garment folds and material properties are preserved through attention to high-level latent codes and spatial consistency.

Experimental results

Research questions

• RQ1 Can latent space interpolation in StyleGAN2 produce realistic, high-resolution virtual try-on results without manual hyperparameter tuning?
• RQ2 How can garment details such as folds, patterns, and material properties be preserved during identity-preserving image synthesis?
• RQ3 To what extent can automatic optimization of interpolation coefficients per latent layer improve realism and consistency in virtual try-on generation?
• RQ4 Can the method generalize across diverse body shapes and garment types while maintaining photo-realism at 512×512 resolution?

Key findings

• The method achieves state-of-the-art photo-realistic results at 512×512 resolution, outperforming prior approaches in visual fidelity.
• Automatic optimization of interpolation coefficients per latent layer enables seamless blending of identity and garment attributes.
• Garment details such as folds, texture, and pattern are preserved effectively due to the disentangled nature of StyleGAN2's latent space.
• The approach successfully deforms garments according to the target person’s body shape while maintaining structural and material realism.