[Paper Review] H-NeRF: Neural Radiance Fields for Rendering and Temporal Reconstruction of Humans in Motion

Hongyi Xu, Thiemo Alldieck+1 more|arXiv (Cornell University)|Oct 26, 2021

3D Shape Modeling and Analysis57 references75 citations

TL;DR

H-NeRF combines neural radiance fields with an implicit human body model (imGHUM) to render and temporally reconstruct humans in motion from sparse views, enabling generalization to novel poses, shapes, and viewpoints.

ABSTRACT

We present neural radiance fields for rendering and temporal (4D) reconstruction of humans in motion (H-NeRF), as captured by a sparse set of cameras or even from a monocular video. Our approach combines ideas from neural scene representation, novel-view synthesis, and implicit statistical geometric human representations, coupled using novel loss functions. Instead of learning a radiance field with a uniform occupancy prior, we constrain it by a structured implicit human body model, represented using signed distance functions. This allows us to robustly fuse information from sparse views and generalize well beyond the poses or views observed in training. Moreover, we apply geometric constraints to co-learn the structure of the observed subject -- including both body and clothing -- and to regularize the radiance field to geometrically plausible solutions. Extensive experiments on multiple datasets demonstrate the robustness and the accuracy of our approach, its generalization capabilities significantly outside a small training set of poses and views, and statistical extrapolation beyond the observed shape.

Motivation & Objective

• Motivate free-viewpoint video of a moving human from sparse camera viewpoints.
• Develop a geometry-aware NeRF regularized by a structured implicit human body model (imGHUM).
• Co-learn a residual SDF with NeRF to capture clothing and hair details beyond the base body model.
• Achieve robust 4D reconstruction and rendering with strong generalization to unseen poses, views, and shapes.

Proposed method

• Co-learn a radiance field Fω that maps x, v to (c, σ) and a residual SDF ΔHω mapping x to Δd, conditioned on body pose θ and root transform T.
• Use imGHUM as a canonical-frame prior to guide geometry and provide spatial-temporal correspondences across frames.
• Couple NeRF with an implicit SDF by blending pseudo alpha from the SDF with NeRF alpha to regularize opacity near the surface.
• Condition NeRF color and geometry on pose codes (β, θ, T) and warp observations to a canonical frame for dynamic sequences.
• Introduce losses L_blend, L_geom, L_seg, and Eikonal-based regularization to enforce surface consistency and geometric plausibility.
• Fine-tune pose/shape corrections Δβ and Δθ(t) during training to improve fit to the foreground.

Experimental results

Research questions

• RQ1 Can a NeRF-based representation fused with an implicit human body model render and reconstruct a moving human from sparse views?
• RQ2 Does co-learning a residual SDF with NeRF improve geometry and appearance, especially for clothing and hair, compared to base NeRF?
• RQ3 How well does the approach generalize to novel poses, shapes, and unseen camera views with limited training data?
• RQ4 What is the impact of pose-conditioned geometry and appearance on rendering fidelity and temporal coherence?
• RQ5 Can the method robustly handle both static and dynamic scenes with sparse camera setups?

Key findings

• H-NeRF delivers superior novel-view rendering and 3D geometry reconstruction under sparse views compared to NeRF and IDR in static scenes.
• For dynamic humans, H-NeRF outperforms NeuralBody on both image quality (PSNR/SSIM/LPIPS) and geometric metrics (Ch, NC, IoU) across multiple datasets.
• H-NeRF generalizes to unseen poses and shapes, with robustness to frame/frame count ablations and the ability to operate with as few as 10 temporal frames per camera in four-camera setups.
• The framework achieves accurate free-viewpoint rendering and 4D reconstruction by conditioning on body latent codes and leveraging imGHUM as a strong geometric prior.
• Quantitative results show improved PSNR/SSIM and competitive or better geometric metrics across RenderPeople, GHS3D, PeopleSnapshot, and Human3.6M datasets.