[Paper Review] High-Resolution Virtual Try-On with Misalignment and Occlusion-Handled Conditions

Sangyun Lee, Gyojung Gu+3 more|arXiv (Cornell University)|Jun 28, 2022

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes a unified try-on condition generator that jointly performs clothing warping and segmentation map generation, enabling misalignment-free, occlusion-aware high-resolution virtual try-on. By fusing features between warping and segmentation modules and introducing discriminator rejection, the method eliminates pixel-squeezing artifacts and achieves state-of-the-art performance on 1024×768 resolution benchmarks.

ABSTRACT

Image-based virtual try-on aims to synthesize an image of a person wearing a given clothing item. To solve the task, the existing methods warp the clothing item to fit the person's body and generate the segmentation map of the person wearing the item before fusing the item with the person. However, when the warping and the segmentation generation stages operate individually without information exchange, the misalignment between the warped clothes and the segmentation map occurs, which leads to the artifacts in the final image. The information disconnection also causes excessive warping near the clothing regions occluded by the body parts, so-called pixel-squeezing artifacts. To settle the issues, we propose a novel try-on condition generator as a unified module of the two stages (i.e., warping and segmentation generation stages). A newly proposed feature fusion block in the condition generator implements the information exchange, and the condition generator does not create any misalignment or pixel-squeezing artifacts. We also introduce discriminator rejection that filters out the incorrect segmentation map predictions and assures the performance of virtual try-on frameworks. Experiments on a high-resolution dataset demonstrate that our model successfully handles the misalignment and occlusion, and significantly outperforms the baselines. Code is available at https://github.com/sangyun884/HR-VITON.

Motivation & Objective

• Address misalignment between warped clothing and segmentation maps in high-resolution virtual try-on.
• Mitigate pixel-squeezing artifacts caused by body-part occlusion of garments.
• Improve performance on high-resolution (1024×768) virtual try-on datasets.
• Enhance real-world applicability by filtering out incorrect segmentation predictions via discriminator rejection.

Proposed method

• Introduce a unified try-on condition generator that jointly predicts warped clothing and segmentation maps, ensuring intrinsic alignment.
• Design a feature fusion block to enable bidirectional information exchange between warping and segmentation modules.
• Implement condition aligning to enforce spatial consistency between warped clothes and predicted segmentation.
• Apply discriminator rejection to filter out low-quality or incorrect segmentation map predictions.
• Use a GAN-based framework with a discriminator that evaluates segmentation map quality to improve final image fidelity.
• Train the model end-to-end on a high-resolution virtual try-on dataset with paired person and clothing images.

Experimental results

Research questions

• RQ1 Can a unified architecture that jointly performs warping and segmentation generation eliminate misalignment artifacts in high-resolution virtual try-on?
• RQ2 How effectively can the proposed method handle occlusion of clothing by body parts, especially in regions with complex geometry?
• RQ3 To what extent does feature fusion between warping and segmentation modules improve visual quality and reduce artifacts?
• RQ4 Can discriminator rejection improve robustness to out-of-distribution inputs in real-world virtual try-on applications?

Key findings

• The proposed method achieves state-of-the-art performance on the 1024×768 resolution virtual try-on benchmark, outperforming VITON-HD and other leading baselines.
• Quantitative results show a KID score of 0.179 and FID of 10.91 in the unpaired setting, significantly better than PF-AFN and other parser-free methods.
• The ablation study confirms that both the feature fusion block and condition aligning are essential, with a 15% improvement in FID when both are used.
• Qualitative results demonstrate complete elimination of misalignment and pixel-squeezing artifacts, especially in occluded regions.
• Discriminator rejection effectively filters out incorrect segmentation predictions, improving robustness in real-world scenarios.
• The model surpasses CP-VTON and ACGPN in high-resolution unpaired settings, where these models fail to generalize.