[Paper Review] HumanNorm: Learning Normal Diffusion Model for High-quality and Realistic 3D Human Generation

Xin Huang, Ruizhi Shao+5 more|arXiv (Cornell University)|Oct 2, 2023

Human Motion and Animation5 citations

TL;DR

HumanNorm proposes a novel normal diffusion model to enhance 2D text-to-image diffusion models for high-quality 3D human generation. By training a normal-adapted diffusion model for geometry and a normal-aligned diffusion model for texture, it generates detailed, realistic 3D humans with improved geometric fidelity and natural appearance, outperforming existing methods in both geometry and texture quality.

ABSTRACT

Recent text-to-3D methods employing diffusion models have made significant advancements in 3D human generation. However, these approaches face challenges due to the limitations of text-to-image diffusion models, which lack an understanding of 3D structures. Consequently, these methods struggle to achieve high-quality human generation, resulting in smooth geometry and cartoon-like appearances. In this paper, we propose HumanNorm, a novel approach for high-quality and realistic 3D human generation. The main idea is to enhance the model's 2D perception of 3D geometry by learning a normal-adapted diffusion model and a normal-aligned diffusion model. The normal-adapted diffusion model can generate high-fidelity normal maps corresponding to user prompts with view-dependent and body-aware text. The normal-aligned diffusion model learns to generate color images aligned with the normal maps, thereby transforming physical geometry details into realistic appearance. Leveraging the proposed normal diffusion model, we devise a progressive geometry generation strategy and a multi-step Score Distillation Sampling (SDS) loss to enhance the performance of 3D human generation. Comprehensive experiments substantiate HumanNorm's ability to generate 3D humans with intricate geometry and realistic appearances. HumanNorm outperforms existing text-to-3D methods in both geometry and texture quality. The project page of HumanNorm is https://humannorm.github.io/.

Motivation & Objective

• Address the limitations of existing text-to-3D methods that rely on text-to-image diffusion models, which lack 3D structural perception and produce smooth, cartoon-like 3D humans.
• Overcome issues such as Janus artifacts, over-saturated textures, and fake 3D details (e.g., in wrinkles and hair) by integrating geometric priors into the generation pipeline.
• Improve 2D diffusion models’ understanding of 3D geometry by introducing a normal-adapted diffusion model trained on multi-view normal maps with view-dependent and body-aware text.
• Enhance texture realism by aligning appearance generation with surface normals through a normal-aligned diffusion model, reducing geometric misalignment and over-saturation.
• Develop a progressive geometry generation strategy and a multi-step Score Distillation Sampling (SDS) loss to stabilize training and improve output quality.

Proposed method

• Train a normal-adapted diffusion model using multi-view normal maps from 3D human scans and prompts with view-dependent and body-aware text to generate high-fidelity, geometry-aware normal maps.
• Introduce a depth-adapted diffusion model to complement normal maps by providing depth information, reducing artifacts in regions with ambiguous normals (e.g., ears).
• Use both normal and depth SDS losses during geometry generation to guide the 3D reconstruction process, ensuring accurate geometric details.
• Implement a progressive geometry generation strategy that gradually refines the 3D mesh, reducing noise and improving stability.
• Train a normal-aligned diffusion model that takes predicted normal maps as conditional input to generate color images aligned with geometric folds and shading.
• Apply a multi-step SDS loss during texture generation to recover images over multiple diffusion steps, mitigating over-saturation and enhancing realism.

Experimental results

Research questions

• RQ1 Can a normal-adapted diffusion model improve 2D diffusion models’ perception of 3D geometry in text-to-3D generation?
• RQ2 How effective is the integration of normal maps and depth maps in reducing geometric artifacts such as Janus faces and smooth surfaces?
• RQ3 Can a normal-aligned diffusion model significantly improve texture realism by aligning appearance with geometric details?
• RQ4 Does a multi-step SDS loss outperform single-step SDS in reducing over-saturation and enhancing texture naturalness?
• RQ5 To what extent does the progressive geometry generation strategy improve the stability and quality of 3D human generation?

Key findings

• HumanNorm generates 3D humans with significantly improved geometric quality, eliminating Janus artifacts and producing detailed facial and clothing geometry.
• The normal-adapted diffusion model successfully generates high-fidelity normal maps from view-dependent and body-aware prompts, enabling accurate 3D geometry reconstruction.
• The depth-adapted diffusion model reduces artifacts in geometrically ambiguous regions (e.g., ears), as shown by ablation studies where depth SDS loss improves geometry quality.
• The normal-aligned diffusion model produces sharper, more realistic textures that align with geometric folds, outperforming standard text-to-image diffusion models in appearance fidelity.
• The multi-step SDS loss effectively reduces over-saturation, producing more natural color and lighting in generated textures compared to single-step SDS.
• HumanNorm achieves superior CLIP scores and user study results, indicating better alignment with text prompts and higher perceived realism in generated 3D humans.