[Paper Review] DeepWrinkles: Accurate and Realistic Clothing Modeling

Zorah Laehner, Daniel Cremers+1 more|arXiv (Cornell University)|Aug 10, 2018

3D Shape Modeling and Analysis46 references42 citations

TL;DR

A data-driven framework with a subspace-based global cloth deformation model and a temporal-consistent cGAN that adds high-frequency wrinkles to normal maps for realistic clothing animation.

ABSTRACT

We present a novel method to generate accurate and realistic clothing deformation from real data capture. Previous methods for realistic cloth modeling mainly rely on intensive computation of physics-based simulation (with numerous heuristic parameters), while models reconstructed from visual observations typically suffer from lack of geometric details. Here, we propose an original framework consisting of two modules that work jointly to represent global shape deformation as well as surface details with high fidelity. Global shape deformations are recovered from a subspace model learned from 3D data of clothed people in motion, while high frequency details are added to normal maps created using a conditional Generative Adversarial Network whose architecture is designed to enforce realism and temporal consistency. This leads to unprecedented high-quality rendering of clothing deformation sequences, where fine wrinkles from (real) high resolution observations can be recovered. In addition, as the model is learned independently from body shape and pose, the framework is suitable for applications that require retargeting (e.g., body animation). Our experiments show original high quality results with a flexible model. We claim an entirely data-driven approach to realistic cloth wrinkle generation is possible.

Motivation & Objective

• Motivate realistic garment reconstruction for animation, AR/VR, and virtual try-on.
• Develop a fully data-driven pipeline that separates body shape/pose from clothing wrinkles.
• Capture high-frequency cloth details from real 4D scans and render them efficiently.
• Enable retargeting of clothing across different body shapes and poses.

Proposed method

• Learn a linear subspace model of clothing deformations from 4D scan data with pose and body-shape normalization.
• Register a clothing template to 4D scans and build a mean shape plus vertex offsets; perform PCA to obtain a low-dimensional representation.
• Train a pose-to-shape regression (and an autoregressive RNN) to map joint poses to subspace coefficients for retargeting.
• Generate fine surface details by training a conditional GAN on normal maps, conditioning on low-resolution normals to produce high-resolution normals.
• Impose temporal consistency through a dedicated loss term to ensure smooth frame-to-frame transitions in the normal maps.
• Render using the enhanced normal maps to achieve realistic wrinkle details in real-time graphics pipelines.

Experimental results

Research questions

• RQ1 Can a data-driven subspace model accurately capture global clothing deformations while factoring out body shape and pose for retargeting?
• RQ2 Can a neural network on normal maps add high-frequency, temporally coherent wrinkles recovered from high-resolution scans?
• RQ3 What is the impact of input normal-map quality and temporal loss on realism and stability of retargeted clothing?
• RQ4 How well does the retargeting generalize to new body shapes and unseen poses?

Key findings

• A linear subspace model learned from real 4D data can represent clothing deformations with pose- and shape-normalized coordinates.
• A cGAN operating on normal maps can recover fine wrinkles not captured by low-resolution templates, producing high-fidelity rendering.
• Incorporating a temporal consistency loss improves frame-to-frame stability and realism in animated sequences.
• Retargeting to new body shapes and poses is feasible, enabling rendering of garments across different characters with preserved detail.
• Compared to physics-based simulations and pure subspace methods, DeepWrinkles achieves higher detail fidelity with a data-driven approach and modest storage.