[Paper Review] HDHumans: A Hybrid Approach for High-fidelity Digital Humans

Marc Habermann, Lingjie Liu+4 more|arXiv (Cornell University)|Oct 21, 2022

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

HDHumans proposes a hybrid framework that tightly integrates a deforming 3D mesh template with a neural radiance field (NeRF) to enable high-fidelity, photo-realistic synthesis of digital humans from multi-view video. By mutually guiding each other—meshes improve NeRF sampling efficiency and generalization, while NeRF enhances surface detail and supervision—the method achieves 4K-resolution, temporally coherent novel view and motion synthesis, even for loose clothing and unseen motions.

ABSTRACT

Photo-real digital human avatars are of enormous importance in graphics, as they enable immersive communication over the globe, improve gaming and entertainment experiences, and can be particularly beneficial for AR and VR settings. However, current avatar generation approaches either fall short in high-fidelity novel view synthesis, generalization to novel motions, reproduction of loose clothing, or they cannot render characters at the high resolution offered by modern displays. To this end, we propose HDHumans, which is the first method for HD human character synthesis that jointly produces an accurate and temporally coherent 3D deforming surface and highly photo-realistic images of arbitrary novel views and of motions not seen at training time. At the technical core, our method tightly integrates a classical deforming character template with neural radiance fields (NeRF). Our method is carefully designed to achieve a synergy between classical surface deformation and NeRF. First, the template guides the NeRF, which allows synthesizing novel views of a highly dynamic and articulated character and even enables the synthesis of novel motions. Second, we also leverage the dense pointclouds resulting from NeRF to further improve the deforming surface via 3D-to-3D supervision. We outperform the state of the art quantitatively and qualitatively in terms of synthesis quality and resolution, as well as the quality of 3D surface reconstruction.

Motivation & Objective

• Address the limitations of existing digital human synthesis methods in handling high-resolution, novel views, unseen motions, and loose clothing.
• Overcome the 3D inconsistency and low fidelity of image-space-only approaches and the limited generalization of mesh-only or NeRF-only methods.
• Enable efficient, high-fidelity synthesis of photo-realistic human avatars from multi-view video using a tightly coupled hybrid representation.
• Achieve state-of-the-art performance in both 3D surface reconstruction and novel view synthesis at 4K resolution.
• Support motion generalization and dynamic clothing deformation without requiring explicit supervision for every motion or garment type.

Proposed method

• Introduce a hybrid representation combining a dense, motion-driven 3D deforming mesh template with a neural radiance field (NeRF) defined in a thin shell around the surface.
• Use the mesh as a geometric prior to guide NeRF sampling and feature aggregation, reducing required samples from 64+128 to 32 per ray and accelerating training by 6×.
• Leverage NeRF’s high-frequency detail and explicit 3D supervision to refine the mesh deformation network via 3D-to-3D supervision from NeRF-estimated points.
• Train the mesh and NeRF jointly using a two-way synergy: mesh guides NeRF for efficient, motion-generalizable rendering; NeRF improves mesh detail and reduces local minima.
• Parameterize the NeRF in a space-time coherent manner, conditioned on skeletal motion and camera pose, enabling novel motion and view synthesis.
• Apply 4K multi-view supervision during training, made feasible by the mesh-guided sampling strategy, to achieve higher resolution output.

Experimental results

Research questions

• RQ1 Can a hybrid mesh-NeRF representation jointly improve 3D surface reconstruction and novel view synthesis quality for high-fidelity digital humans?
• RQ2 How can a deforming mesh template be used to guide NeRF sampling and improve efficiency and generalization to unseen motions and poses?
• RQ3 To what extent can NeRF supervision enhance the accuracy and detail of a learned deforming mesh, especially for loose clothing and high-frequency geometry?
• RQ4 Can this hybrid approach achieve 4K-resolution photo-realistic rendering while maintaining temporal coherence and motion generalization?
• RQ5 What are the practical limits of training efficiency and inference speed in such a tightly coupled system, and how can they be improved?

Key findings

• HDHumans achieves state-of-the-art performance in both 3D surface reconstruction and novel view synthesis, with significantly sharper details and reduced geometric artifacts.
• The method reduces NeRF sampling from 64+128 to 32 samples per ray, cutting training time from 29 days to 10 days on 4K multi-view videos.
• Quantitative evaluation on 4112×3008 resolution images confirms that 4K supervision leads to lower reconstruction error and improved fidelity.
• The approach successfully synthesizes photo-realistic images of novel views and unseen motions, including complex dynamic clothing like skirts.
• Motion retargeting results show high realism, even transferring motion from a blue dress to a yellow skirt while preserving fine details like wrinkles.
• Video synthesis applications demonstrate the method’s ability to seamlessly overlay photo-realistic avatars into dynamic scenes with consistent lighting and geometry.