[Paper Review] ARCH: Animatable Reconstruction of Clothed Humans

Zeng Huang, Yuanlu Xu+3 more|arXiv (Cornell University)|Apr 8, 2020

3D Shape Modeling and Analysis55 references12 citations

TL;DR

ARCH proposes an end-to-end framework for reconstructing detailed, animation-ready 3D clothed human avatars from a single monocular RGB image. By leveraging a learned semantic space and deformation field to normalize pose variations, and using opacity-aware differentiable rendering for fine-grained refinement, ARCH achieves over 50% lower reconstruction error than state-of-the-art methods on public benchmarks while producing high-fidelity, rigged models ready for animation.

ABSTRACT

In this paper, we propose ARCH (Animatable Reconstruction of Clothed Humans), a novel end-to-end framework for accurate reconstruction of animation-ready 3D clothed humans from a monocular image. Existing approaches to digitize 3D humans struggle to handle pose variations and recover details. Also, they do not produce models that are animation ready. In contrast, ARCH is a learned pose-aware model that produces detailed 3D rigged full-body human avatars from a single unconstrained RGB image. A Semantic Space and a Semantic Deformation Field are created using a parametric 3D body estimator. They allow the transformation of 2D/3D clothed humans into a canonical space, reducing ambiguities in geometry caused by pose variations and occlusions in training data. Detailed surface geometry and appearance are learned using an implicit function representation with spatial local features. Furthermore, we propose additional per-pixel supervision on the 3D reconstruction using opacity-aware differentiable rendering. Our experiments indicate that ARCH increases the fidelity of the reconstructed humans. We obtain more than 50% lower reconstruction errors for standard metrics compared to state-of-the-art methods on public datasets. We also show numerous qualitative examples of animated, high-quality reconstructed avatars unseen in the literature so far.

Motivation & Objective

• To address the challenge of reconstructing detailed, 3D, animatable clothed human avatars from unconstrained monocular images.
• To overcome limitations in existing methods that fail to handle pose variations, occlusions, and lack rigging for animation.
• To produce geometry and appearance representations that are both highly detailed and compatible with standard animation pipelines.
• To improve reconstruction fidelity through pose-aware normalization and per-pixel supervision via differentiable rendering.
• To enable direct animation of reconstructed avatars using motion capture data without additional rigging.

Proposed method

• Introduces a Semantic Space (SemS) and Semantic Deformation Field (SemDF) to map 3D clothed humans into a canonical pose space, reducing ambiguity from pose and occlusion.
• Employs an implicit neural representation with spatial local features (e.g., RBF-based joint distances) to model high-frequency surface details like wrinkles and clothing folds.
• Uses a differentiable renderer with opacity-aware Granular Render-and-Compare to refine predicted normals and albedo by comparing rendered images to the input.
• Leverages predicted 3D body shape and pose from a 2D image to initialize a canonical-space reconstruction with skinning weights for automatic rigging.
• Applies per-pixel supervision on normals and albedo during training to enhance surface detail and relighting fidelity.
• Constructs a canonical-space mesh that is inherently rigged, enabling direct use in animation pipelines.

Experimental results

Research questions

• RQ1 Can a single-image 3D reconstruction method produce high-fidelity, animatable clothed human avatars with detailed geometry and appearance?
• RQ2 How can pose variations and occlusions in unconstrained images be effectively normalized to improve reconstruction accuracy?
• RQ3 Can differentiable rendering with opacity-aware supervision enhance the realism of normals and albedo in implicit 3D representations?
• RQ4 To what extent can implicit functions with semantic deformation fields reconstruct fine details like clothing wrinkles and hair?
• RQ5 Can reconstructed avatars be directly animated using standard motion capture sequences without additional processing?

Key findings

• ARCH reduces reconstruction error by over 50% on the RenderPeople dataset and more than 60% on the BUFF dataset compared to state-of-the-art methods like PIFu.
• The RBF-based spatial feature representation outperforms XYZ and L2 features in both quantitative metrics and qualitative detail preservation.
• Granular Render-and-Compare with opacity-aware differentiable rendering significantly improves normal estimation and reduces visual artifacts.
• The framework successfully reconstructs complex clothing details such as pants seams, shoe heels, and fine wrinkles not captured by prior methods.
• Reconstructed avatars are inherently rigged and can be directly animated using retargeted Mixamo motion sequences, demonstrating full animation readiness.
• The model generalizes well to challenging poses and self-contact scenarios, though rare or heavily occluded poses remain a challenge.