[论文解读] SPG-VTON: Semantic Prediction Guidance for Multi-pose Virtual Try-on

Bingwen Hu, Liu Ping+2 more|arXiv (Cornell University)|Aug 3, 2021

Generative Adversarial Networks and Image Synthesis参考文献 49被引用 4

一句话总结

SPG-VTON 提出了一种用于多姿态虚拟试穿的端到端语义预测引导网络，通过语义预测模块（SPM）、带有可传导循环一致性损失的服装形变模块（CWM）以及虚拟试穿生成模块（TSM），生成逼真的结果。在 MPV 和 DeepFashion 数据集上，其性能优于当前最先进方法，对姿态变化、背景噪声和配饰具有鲁棒性。

ABSTRACT

Image-based virtual try-on is challenging in fitting a target in-shop clothes into a reference person under diverse human poses. Previous works focus on preserving clothing details ( e.g., texture, logos, patterns ) when transferring desired clothes onto a target person under a fixed pose. However, the performances of existing methods significantly dropped when extending existing methods to multi-pose virtual try-on. In this paper, we propose an end-to-end Semantic Prediction Guidance multi-pose Virtual Try-On Network (SPG-VTON), which could fit the desired clothing into a reference person under arbitrary poses. Concretely, SPG-VTON is composed of three sub-modules. First, a Semantic Prediction Module (SPM) generates the desired semantic map. The predicted semantic map provides more abundant guidance to locate the desired clothes region and produce a coarse try-on image. Second, a Clothes Warping Module (CWM) warps in-shop clothes to the desired shape according to the predicted semantic map and the desired pose. Specifically, we introduce a conductible cycle consistency loss to alleviate the misalignment in the clothes warping process. Third, a Try-on Synthesis Module (TSM) combines the coarse result and the warped clothes to generate the final virtual try-on image, preserving details of the desired clothes and under the desired pose. Besides, we introduce a face identity loss to refine the facial appearance and maintain the identity of the final virtual try-on result at the same time. We evaluate the proposed method on the most massive multi-pose dataset (MPV) and the DeepFashion dataset. The qualitative and quantitative experiments show that SPG-VTON is superior to the state-of-the-art methods and is robust to the data noise, including background and accessory changes, i.e., hats and handbags, showing good scalability to the real-world scenario.

研究动机与目标

• 为解决现有多姿态虚拟试穿方法存在的对齐错误、细节保留差以及缺乏端到端训练的问题。
• 通过利用语义先验知识，提升在任意人体姿态间转移服装时的贴合精度与真实感。
• 通过实现端到端训练与推理，减少误差累积与人工干预。
• 通过专用的面部身份损失，保持面部身份一致性和外观真实感。
• 通过增强对背景杂乱和配饰（如帽子、手提包）等噪声数据的鲁棒性。

提出的方法

• 语义预测模块（SPM）为目标人物生成预测的语义图，为服装区域定位提供空间与语义引导。
• 服装形变模块（CWM）基于预测的语义图和目标姿态对店内服装进行形变，使用可传导的循环一致性损失以减少形变错位。
• 虚拟试穿生成模块（TSM）融合形变后的服装与粗略结果，生成最终的逼真试穿图像，通过全局与局部判别器确保形状与纹理保真度。
• 应用面部身份损失以保持面部身份并提升生成面部区域的真实感。
• 整个网络通过五个超参数端到端训练，用于平衡重建损失、感知损失、循环一致性损失与身份损失。
• 可传导的循环一致性损失确保形变服装与目标身体形状之间的结构一致性，从而提升形变精度。

Figure 1: Visual results of multi-pose virtual try-on by the proposed method. First column: source images. Top row: in-shop clothes and target poses. Other columns: generated images. SPG-VTON produces photo-realistic virtual try-on results, which preserves both clothes details and person characteris

实验结果

研究问题

• RQ1 与多阶段基线相比，采用语义预测的端到端框架是否能显著提升多姿态虚拟试穿性能？
• RQ2 在任意姿态下，可传导的循环一致性损失在减少服装形变错位方面效果如何？
• RQ3 语义图预测精度在多大程度上影响最终试穿质量？
• RQ4 在姿态变化和噪声输入下，模型能否保持面部身份与真实感？
• RQ5 该方法对背景杂乱和帽子、手提包等配饰的鲁棒性如何？

主要发现

• SPG-VTON 在 MPV 和 DeepFashion 数据集上达到最先进性能，其 SSIM/Mask-SSIM 和 Mask-IS 得分在对比方法中最高。
• 消融实验表明，若移除任意一个超参数（λ₁ 至 λ₅），性能均显著下降，证实其必要性。
• 即使在噪声数据上进行训练，模型仍能保持高质量结果，仅出现轻微指标下降，且对未见的噪声测试图像泛化能力良好。
• 在推理时使用真实语义图的得分高于使用预测语义图，但端到端训练提升了预测语义图的准确性，从而缩小了两者差距。
• 可传导的循环一致性损失至关重要——若移除该损失，输出中服装细节将完全丢失。
• 面部身份损失显著提升了面部真实感，消融实验表明禁用该损失后面部区域显得更不自然。

Figure 2: The overview of our SPG-VTON. (a) The Semantic Prediction Module (SPM) consists of two processes. One is the target semantic map prediction, and the other one is the target clothes mask prediction and coarse result generation. (b) The Clothes Warping Module (CWM) warps the in-shop clothes