[Paper Review] HumanGaussian: Text-Driven 3D Human Generation with Gaussian Splatting

Xian Liu, Xiaohang Zhan+6 more|arXiv (Cornell University)|Nov 28, 2023

3D Shape Modeling and Analysis4 citations

TL;DR

HumanGaussian proposes a text-to-3D human generation framework using 3D Gaussian Splatting (3DGS) with structure-aware score distillation sampling (SDS) and annealed negative prompt guidance. By initializing Gaussians on SMPL-X mesh and leveraging dual-branch (RGB + depth) SDS, it achieves high-fidelity, detailed 3D humans with realistic appearance and fine-grained geometry—outperforming prior methods in quality and efficiency while avoiding over-saturation at low classifier-free guidance (CFG) scales.

ABSTRACT

Realistic 3D human generation from text prompts is a desirable yet challenging task. Existing methods optimize 3D representations like mesh or neural fields via score distillation sampling (SDS), which suffers from inadequate fine details or excessive training time. In this paper, we propose an efficient yet effective framework, HumanGaussian, that generates high-quality 3D humans with fine-grained geometry and realistic appearance. Our key insight is that 3D Gaussian Splatting is an efficient renderer with periodic Gaussian shrinkage or growing, where such adaptive density control can be naturally guided by intrinsic human structures. Specifically, 1) we first propose a Structure-Aware SDS that simultaneously optimizes human appearance and geometry. The multi-modal score function from both RGB and depth space is leveraged to distill the Gaussian densification and pruning process. 2) Moreover, we devise an Annealed Negative Prompt Guidance by decomposing SDS into a noisier generative score and a cleaner classifier score, which well addresses the over-saturation issue. The floating artifacts are further eliminated based on Gaussian size in a prune-only phase to enhance generation smoothness. Extensive experiments demonstrate the superior efficiency and competitive quality of our framework, rendering vivid 3D humans under diverse scenarios. Project Page: https://alvinliu0.github.io/projects/HumanGaussian

Motivation & Objective

• To address the challenge of generating high-fidelity 3D humans with fine-grained geometry and realistic appearance from text prompts.
• To overcome limitations in existing 3D generation methods based on mesh or neural fields, which suffer from poor detail capture or high training cost.
• To adapt 3D Gaussian Splatting—previously used for reconstruction—into a text-driven 3D generation framework with improved optimization stability and structural fidelity.
• To mitigate over-saturation and floating artifacts in diffusion-based 3D generation by introducing annealed negative prompt guidance and size-conditioned pruning.
• To enable efficient, controllable 3D human generation using only text prompts, without requiring multi-view supervision or complex optimization pipelines.

Proposed method

• Structure-Aware SDS initializes Gaussians on the SMPL-X mesh and jointly optimizes appearance and geometry using a multi-modal score function from both RGB and depth spaces.
• Dual-branch SDS extends the pre-trained Stable Diffusion model to denoise both RGB and depth images simultaneously, enabling joint texture and structural guidance during optimization.
• Annealed Negative Prompt Guidance decomposes the SDS loss into a noisier generative score and a cleaner classifier score, with the latter gradually reduced during training to avoid over-saturation at low CFG scales (e.g., 7.5).
• Size-conditioned Gaussian pruning is applied in a separate prune-only phase to remove tiny, floating artifacts by eliminating Gaussians below a size threshold.
• The framework leverages 3DGS’s efficient rasterization and adaptive density control, where densification and pruning are guided by intrinsic human body structure and multi-view consistency.
• The method integrates human priors (SMPL-X and skeleton pose) directly into the SDS optimization process to improve geometric coherence and reduce artifacts.

Figure 2 : Overview of the proposed HumanGaussian Framework. We generate high-quality 3D humans from text prompts with the neural representation of 3D Gaussian Splatting (3DGS). In Structure-Aware SDS , we start from the SMPL-X prior to densely sample Gaussians on the human mesh surface as initial c

Experimental results

Research questions

• RQ1 Can 3D Gaussian Splatting be effectively adapted for text-driven 3D human generation with high geometric and appearance fidelity?
• RQ2 How can structural priors such as SMPL-X and pose guidance improve optimization stability and detail capture in 3DGS-based generation?
• RQ3 Can annealed negative prompt guidance reduce over-saturation and improve realism at low classifier-free guidance (CFG) scales?
• RQ4 How does dual-branch (RGB + depth) SDS improve geometric consistency and texture realism compared to single-branch supervision?
• RQ5 To what extent can size-conditioned pruning eliminate floating artifacts in 3DGS-based 3D generation?

Key findings

• HumanGaussian achieves the highest mean opinion score (MOS) of 4.24 in texture quality, 3.88 in geometry quality, and 4.71 in text alignment across 17 user evaluators, significantly outperforming prior methods.
• The method reduces over-saturation issues by using annealed negative prompt guidance, enabling high-quality results at a CFG scale of 7.5 instead of requiring large values like 100.
• Size-conditioned pruning effectively eliminates floating artifacts near the human surface, improving visual smoothness and realism in the final 3D output.
• Dual-branch SDS (RGB + depth) leads to better geometric regularization, especially in challenging regions like limbs and hair, compared to single-branch supervision.
• The integration of SMPL-X as an initialization prior significantly improves structural coherence and reduces the multi-face Janus problem seen in baseline 3DGS generation.
• HumanGaussian demonstrates superior efficiency and quality compared to 3DGS-based baselines and prior SOTA methods like DreamHuman and TADA, even when those baselines are augmented with multi-view or shape priors.

Figure 3 : Visual Comparisons with Text-to-3D and 3D Human Models. We compare with recent state-of-the-art baselines on five different prompts, each showing two camera views. Note that the textural unrealism and blurriness are highlighted with yellow arrows; the geometric artifacts are highlighted w