[Paper Review] Towards Scalable Unpaired Virtual Try-On via Patch-Routed Spatially-Adaptive GAN

Zhenyu Xie, Zaiyu Huang+4 more|arXiv (Cornell University)|Nov 20, 2021

3D Shape Modeling and Analysis19 citations

TL;DR

This paper proposes PASTA-GAN, a scalable unpaired virtual try-on framework that disentangles garment style and spatial features using a patch-routed disentanglement module and spatially-adaptive residual blocks, enabling high-fidelity, end-to-end garment transfer without paired data or online optimization, achieving state-of-the-art performance on MPV and generalizing to diverse garments including full-body and rare types like slings.

ABSTRACT

Image-based virtual try-on is one of the most promising applications of human-centric image generation due to its tremendous real-world potential. Yet, as most try-on approaches fit in-shop garments onto a target person, they require the laborious and restrictive construction of a paired training dataset, severely limiting their scalability. While a few recent works attempt to transfer garments directly from one person to another, alleviating the need to collect paired datasets, their performance is impacted by the lack of paired (supervised) information. In particular, disentangling style and spatial information of the garment becomes a challenge, which existing methods either address by requiring auxiliary data or extensive online optimization procedures, thereby still inhibiting their scalability. To achieve a \emph{scalable} virtual try-on system that can transfer arbitrary garments between a source and a target person in an unsupervised manner, we thus propose a texture-preserving end-to-end network, the PAtch-routed SpaTially-Adaptive GAN (PASTA-GAN), that facilitates real-world unpaired virtual try-on. Specifically, to disentangle the style and spatial information of each garment, PASTA-GAN consists of an innovative patch-routed disentanglement module for successfully retaining garment texture and shape characteristics. Guided by the source person keypoints, the patch-routed disentanglement module first decouples garments into normalized patches, thus eliminating the inherent spatial information of the garment, and then reconstructs the normalized patches to the warped garment complying with the target person pose. Given the warped garment, PASTA-GAN further introduces novel spatially-adaptive residual blocks that guide the generator to synthesize more realistic garment details.

Motivation & Objective

• To address the scalability limitations of existing virtual try-on methods that rely on paired training data, which are labor-intensive and restrictive.
• To overcome the challenge of entangled garment style and spatial features in unpaired settings, which degrades texture and shape fidelity.
• To eliminate dependency on auxiliary data or online optimization procedures while maintaining high-quality synthesis.
• To enable end-to-end, real-world applicable virtual try-on for arbitrary garments across upper body, lower body, and full-body scenarios.

Proposed method

• Introduces a patch-routed disentanglement module that decomposes garments into normalized spatial patches, reducing inherent spatial information to encourage spatial-agnostic style learning.
• Uses source person keypoint guidance to reconstruct normalized patches into a warped garment that conforms to the target person's pose, avoiding explicit warping networks.
• Employs a novel spatially-adaptive residual block that dynamically adjusts feature alignment, correcting misalignment between warped garment and target body shape.
• Integrates warped garment features into intermediate layers of the generator to guide realistic texture synthesis and reduce artifacts.
• Combines style and texture synthesis branches to preserve both garment identity and fine-grained details during generation.
• Trains the model end-to-end in an unsupervised manner using reconstruction loss, avoiding the need for paired supervision or 3D human models.

Experimental results

Research questions

• RQ1 Can a fully unsupervised virtual try-on model achieve high-fidelity garment transfer without paired training data or auxiliary supervision?
• RQ2 How can garment style and spatial features be effectively disentangled in the absence of paired supervision to preserve texture and shape?
• RQ3 Can patch-based decomposition and reconstruction improve garment fidelity compared to global warping or flow-based methods?
• RQ4 To what extent does the spatially-adaptive residual module mitigate feature misalignment in complex pose and shape variations?
• RQ5 Does the proposed method generalize to diverse garments, including rare or full-body types, under unpaired settings?

Key findings

• PASTA-GAN outperforms state-of-the-art paired methods, including PFAFN, on the MPV benchmark in both FID (22.4) and human evaluation scores, demonstrating superiority in unpaired settings.
• The model achieves high-quality results on challenging cases such as full-body try-ons and rare garments like slings, which paired methods fail to handle.
• Abllation studies confirm that the patch-routed disentanglement module is essential for accurate garment shape synthesis, while the spatially-adaptive residual block is critical for texture clarity and artifact reduction.
• Removing the texture synthesis branch leads to poor texture preservation, while replacing spatially-adaptive blocks with normal ones causes misalignment and blurring.
• The full model configuration achieves the best visual and quantitative results, with significant improvements in FID and human evaluation over ablated variants.
• The method generalizes effectively across diverse garments and poses without requiring online optimization or auxiliary data, confirming its scalability.