[Paper Review] Dressing 3D Humans using a Conditional Mesh-VAE-GAN.

Qianli Ma, Siyu Tang+4 more|arXiv (Cornell University)|Jul 31, 2019

3D Shape Modeling and Analysis56 references9 citations

TL;DR

This paper proposes a conditional Mesh-VAE-GAN that generates diverse, realistic 3D clothing on SMPL body meshes by learning from 3D scans of dressed humans. It conditions on clothing type and pose, enabling synthesis of varied garments in novel poses while preserving global shape and local structure, marking the first conditional VAE-GAN for 3D meshes and the first direct 3D mesh dressing model for generalizable clothing generation.

ABSTRACT

Three-dimensional human body models are widely used in the analysis of human pose and motion. Existing models, however, are learned from minimally-clothed humans and thus do not capture the complexity of dressed humans in common images and videos. To address this, we learn a generative 3D mesh model of clothing from 3D scans of people with varying pose. Going beyond previous work, our generative model is conditioned on different clothing types, giving the ability to dress different body shapes in a variety of clothing. To do so, we train a conditional Mesh-VAE-GAN on clothing displacements from a 3D SMPL body model. This generative clothing model enables us to sample various types of clothing, in novel poses, on top of SMPL. With a focus on clothing geometry, the model captures both global shape and local structure, effectively extending the SMPL model to add clothing. To our knowledge, this is the first conditional VAE-GAN that works on 3D meshes. For clothing specifically, it is the first such model that directly dresses 3D human body meshes and generalizes to different poses.

Motivation & Objective

• To address the lack of 3D human body models that capture realistic clothing from common images and videos.
• To learn a generative 3D mesh model of clothing that generalizes across different body shapes and poses.
• To develop a conditional generative model that can synthesize various clothing types on SMPL meshes.
• To extend the SMPL model with realistic, geometry-aware clothing by modeling clothing displacements from a base body shape.
• To create the first conditional VAE-GAN framework tailored for 3D mesh data, enabling controlled and diverse clothing generation.

Proposed method

• The model is trained on 3D scans of dressed humans to learn clothing displacements relative to a SMPL body model.
• A conditional Mesh-VAE-GAN architecture is used, where the generator learns to produce 3D mesh deformations conditioned on clothing type and pose.
• The VAE component learns a disentangled latent representation of clothing geometry, capturing both global shape and local details.
• The GAN component improves realism by adversarially training the generator against a discriminator that distinguishes real from generated clothing meshes.
• The model conditions on SMPL body parameters (shape and pose) and clothing category to generate diverse, pose-generalizable clothing.
• The framework directly outputs 3D mesh clothing that can be overlaid on SMPL bodies in novel poses without retraining.

Experimental results

Research questions

• RQ1 Can a conditional generative model learn to produce diverse, realistic 3D clothing on SMPL body meshes across varying poses?
• RQ2 Can a VAE-GAN framework be effectively adapted to 3D mesh data to model complex clothing geometry and structure?
• RQ3 Does conditioning on clothing type and body pose enable generalization to unseen combinations of body shapes and garments?
• RQ4 Can the model generate clothing that preserves both global form and local details such as folds and draping?
• RQ5 Is it possible to extend the SMPL model with a differentiable, controllable clothing generation mechanism using 3D mesh-based generative modeling?

Key findings

• The proposed conditional Mesh-VAE-GAN successfully generates diverse 3D clothing on SMPL bodies across a range of poses and clothing types.
• The model captures both global garment shape and fine-grained local structures such as folds and creases, improving realism.
• The framework generalizes to novel body shapes and poses not seen during training, demonstrating strong zero-shot generalization.
• The model is the first conditional VAE-GAN designed for 3D mesh data, enabling controlled and disentangled clothing generation.
• The approach directly dresses 3D human meshes without requiring retraining or optimization per new pose, enabling efficient inference.
• The method effectively extends the SMPL model with a realistic, geometry-aware clothing component that preserves structural fidelity.