[Paper Review] HandNeRF: Neural Radiance Fields for Animatable Interacting Hands

Zhiyang Guo, Wengang Zhou+3 more|arXiv (Cornell University)|Mar 24, 2023

Human Pose and Action Recognition4 citations

TL;DR

HandNeRF proposes a unified neural radiance field framework for high-fidelity, photo-realistic reconstruction and novel view synthesis of single or interacting hands across diverse poses. By leveraging a pose-driven deformation field to map to a canonical space, depth-guided density optimization, and neural feature distillation, it achieves state-of-the-art results on the InterHand2.6M dataset with improved geometry, texture, and occlusion handling.

ABSTRACT

We propose a novel framework to reconstruct accurate appearance and geometry with neural radiance fields (NeRF) for interacting hands, enabling the rendering of photo-realistic images and videos for gesture animation from arbitrary views. Given multi-view images of a single hand or interacting hands, an off-the-shelf skeleton estimator is first employed to parameterize the hand poses. Then we design a pose-driven deformation field to establish correspondence from those different poses to a shared canonical space, where a pose-disentangled NeRF for one hand is optimized. Such unified modeling efficiently complements the geometry and texture cues in rarely-observed areas for both hands. Meanwhile, we further leverage the pose priors to generate pseudo depth maps as guidance for occlusion-aware density learning. Moreover, a neural feature distillation method is proposed to achieve cross-domain alignment for color optimization. We conduct extensive experiments to verify the merits of our proposed HandNeRF and report a series of state-of-the-art results both qualitatively and quantitatively on the large-scale InterHand2.6M dataset.

Motivation & Objective

• To address the challenge of reconstructing accurate, photo-realistic geometry and appearance for interacting hands with complex self-occlusions and limited view coverage.
• To enable high-fidelity novel view and novel pose synthesis for gesture animation and virtual reality applications.
• To unify single-hand and interacting-hand modeling within a single NeRF-based framework using pose-aware deformation and cross-domain feature alignment.
• To overcome limitations of traditional mesh-based models in capturing high-frequency details and handling occluded regions in hand interactions.
• To improve robustness in density prediction and texture learning using depth supervision and neural feature distillation from a 2D teacher network.

Proposed method

• A pose-conditioned deformation field warps rays from different hand poses into a shared canonical space, enabling unified NeRF optimization.
• A learnable error-correction network enhances non-rigid deformation modeling beyond standard blend skinning.
• Pseudo depth maps derived from human pose priors are used as supervision to guide occlusion-aware density learning and improve geometric accuracy.
• Neural feature distillation aligns features from a pre-trained 2D teacher network with the 3D color field, enhancing texture quality and reducing artifacts.
• A shared canonical NeRF models both geometry and appearance across all poses, with multi-view consistent volume rendering for novel view synthesis.
• The framework supports both single-hand and two-interacting-hands scenarios through a unified ray composition and mapping strategy.

Experimental results

Research questions

• RQ1 Can a unified NeRF framework effectively model both single and interacting hands with high geometric and textural fidelity across diverse poses?
• RQ2 How can depth supervision derived from human pose priors improve density estimation and occlusion handling in low-visibility regions?
• RQ3 To what extent does neural feature distillation from a 2D teacher network enhance texture quality and reduce artifacts in 3D hand reconstruction?
• RQ4 How does the pose-driven deformation field improve generalization and consistency across novel poses and views?
• RQ5 What is the relative contribution of depth supervision and feature distillation to overall performance on challenging hand interaction scenarios?

Key findings

• HandNeRF achieves a PSNR of 33.0204 on single-hand reconstruction with the full pipeline, significantly outperforming ablations without distillation (32.8421) or depth supervision (30.1057).
• The ablation study shows that depth supervision reduces depth error (DE) from 0.2106 to 0.1840, indicating improved geometric consistency in novel view rendering.
• Neural feature distillation reduces LPIPS from 0.0488 (w/o distillation) to 0.0475, demonstrating improved perceptual quality and texture realism.
• The TransRenderer-based neural renderer achieves the lowest LPIPS (0.0479) and DE (0.1364), indicating better preservation of fine details and depth accuracy.
• The model produces high-fidelity, photo-realistic images and videos of interacting hands from arbitrary views, validated both qualitatively and quantitatively on the large-scale InterHand2.6M benchmark.
• The framework is compatible with both single-hand and two-interacting-hands scenarios, enabling consistent and robust performance across diverse interaction postures.