[Paper Review] Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On

Igor Santesteban, Nils Thuerey+2 more|arXiv (Cornell University)|May 13, 2021

3D Shape Modeling and Analysis73 references5 citations

TL;DR

This paper proposes a self-supervised, data-driven generative model for 3D virtual try-on that eliminates garment-body collisions without postprocessing by introducing a novel canonical space for garments and a diffused human body representation. The method learns collision-free deformations through a self-supervised loss in a canonical deformation subspace, achieving state-of-the-art results on unseen body shapes and motions with 0.09% collision rate.

ABSTRACT

We propose a new generative model for 3D garment deformations that enables us to learn, for the first time, a data-driven method for virtual try-on that effectively addresses garment-body collisions. In contrast to existing methods that require an undesirable postprocessing step to fix garment-body interpenetrations at test time, our approach directly outputs 3D garment configurations that do not collide with the underlying body. Key to our success is a new canonical space for garments that removes pose-and-shape deformations already captured by a new diffused human body model, which extrapolates body surface properties such as skinning weights and blendshapes to any 3D point. We leverage this representation to train a generative model with a novel self-supervised collision term that learns to reliably solve garment-body interpenetrations. We extensively evaluate and compare our results with recently proposed data-driven methods, and show that our method is the first to successfully address garment-body contact in unseen body shapes and motions, without compromising realism and detail.

Motivation & Objective

• To address the persistent problem of garment-body interpenetrations in data-driven virtual try-on, which commonly require postprocessing in existing methods.
• To eliminate the need for postprocessing by learning collision-free garment deformations directly during training.
• To generalize robustly to unseen body shapes and motions, ensuring realism and physical plausibility.
• To develop a differentiable, canonical representation of garment deformations that enables self-supervised collision learning.
• To improve inference efficiency and realism while maintaining high geometric detail in generated garments.

Proposed method

• Introduces a diffused, volumetric representation of the human body that extends skinning weights and blendshapes to any 3D point, enabling smooth deformation modeling.
• Proposes a novel canonical space for garments that removes pose- and shape-dependent deformations already captured by the diffused body model.
• Defines skinning parameters as functions of unposed and deshaped garment configurations, ensuring collision-free projection to the posed state.
• Leverages a self-supervised collision loss by exhaustively sampling the generative deformation subspace to learn collision avoidance.
• Uses a regressor to predict deformations in the canonical space, followed by a differentiable projection to the final 3D mesh using the garment model.
• Employs a generative model trained end-to-end with a novel loss that penalizes interpenetrations in the canonical space, ensuring collision-free outputs.

Experimental results

Research questions

• RQ1 Can a data-driven 3D virtual try-on method learn to avoid garment-body collisions without postprocessing?
• RQ2 How can a canonical garment representation be constructed to decouple shape and pose deformations from the underlying body model?
• RQ3 Can self-supervised learning effectively train a generative model to avoid collisions across diverse body shapes and motions?
• RQ4 To what extent can the method generalize to unseen body shapes and dynamic motions not seen during training?
• RQ5 How does the proposed method compare in performance and realism to physics-based simulation and existing data-driven approaches?

Key findings

• The proposed method achieves a collision rate of only 0.09% on unseen test sequences from the AMASS dataset, significantly outperforming state-of-the-art methods.
• Without the full collision loss, the method's collision rate increases to 0.62%, demonstrating the critical role of the self-supervised loss term.
• The method generalizes effectively to unseen body shapes, including extreme morphologies far beyond the training distribution, with no visible interpenetrations.
• Even in highly dynamic sequences like a flowing dress, the method produces realistic wrinkles and dynamics comparable to physics-based simulation.
• Runtime performance is efficient, with inference times under 3ms per frame for complex garments (e.g., 23k triangles), enabling real-time applications.
• Postprocessing-based methods, while reducing collisions, often introduce new artifacts such as unnatural bulges, which the proposed method avoids entirely.