[Paper Review] Generalizable Neural Performer: Learning Robust Radiance Fields for Human Novel View Synthesis

Wei Cheng, Xu Su+5 more|arXiv (Cornell University)|Apr 25, 2022

3D Shape Modeling and Analysis20 citations

TL;DR

The paper introduces Generalizable Neural Performer (GNR), a generalizable implicit radiance-field framework for free-viewpoint human synthesis from sparse views, combining implicit geometric body embedding and screen-space occlusion-aware appearance blending to achieve robust cross-subject and cross-pose rendering without per-case fine-tuning. It also presents the GeneBody-1.0 dataset for robust evaluation.

ABSTRACT

This work targets at using a general deep learning framework to synthesize free-viewpoint images of arbitrary human performers, only requiring a sparse number of camera views as inputs and skirting per-case fine-tuning. The large variation of geometry and appearance, caused by articulated body poses, shapes and clothing types, are the key bottlenecks of this task. To overcome these challenges, we present a simple yet powerful framework, named Generalizable Neural Performer (GNR), that learns a generalizable and robust neural body representation over various geometry and appearance. Specifically, we compress the light fields for novel view human rendering as conditional implicit neural radiance fields from both geometry and appearance aspects. We first introduce an Implicit Geometric Body Embedding strategy to enhance the robustness based on both parametric 3D human body model and multi-view images hints. We further propose a Screen-Space Occlusion-Aware Appearance Blending technique to preserve the high-quality appearance, through interpolating source view appearance to the radiance fields with a relax but approximate geometric guidance. To evaluate our method, we present our ongoing effort of constructing a dataset with remarkable complexity and diversity. The dataset GeneBody-1.0, includes over 360M frames of 370 subjects under multi-view cameras capturing, performing a large variety of pose actions, along with diverse body shapes, clothing, accessories and hairdos. Experiments on GeneBody-1.0 and ZJU-Mocap show better robustness of our methods than recent state-of-the-art generalizable methods among all cross-dataset, unseen subjects and unseen poses settings. We also demonstrate the competitiveness of our model compared with cutting-edge case-specific ones. Dataset, code and model will be made publicly available.

Motivation & Objective

• Address the challenge of synthesizing free-viewpoint images of arbitrary human performers from sparse multi-view inputs without per-case fine-tuning.
• Develop a robust, generalizable neural body representation that handles pose, shape, and clothing variations.
• Incorporate geometric priors and source-view appearance cues to improve geometry fidelity and appearance realism across views.
• Provide a diverse multi-view dataset (GeneBody-1.0) to benchmark generalizable human rendering.

Proposed method

• Introduce Implicit Geometric Body Embedding to condition the neural radiance field on SMPLx-based geometry and multi-view cues.
• Use a signed distance function (SDF) based on the SMPLx surface plus canonical-space semantic embedding to anchor the body geometry.
• Extract and fuse multi-view image features with the body embedding to inform the radiance field in a conditional NeRF framework.
• Propose Screen-Space Occlusion-Aware Appearance Blending (SSOA-AB) that decomposes blending into an occlusion map and a view-attention-based blending, allowing source-view textures to rectify radiance predictions while mitigating ghosting in unobserved regions.
• Render via a conditional NeRF with volume rendering, training with photometric and geometric losses (including occupancy and occlusion supervision when 3D ground truth is available).
• Train and evaluate on GeneBody-1.0 and ZJU-Mocap, comparing to generalization baselines (pixelNerf, IBRNet) and case-specific methods (NeuralBody, NeuralTexture, NHR, NeuralVolumes).

Experimental results

Research questions

• RQ1 Can a single model learn a generalizable and robust implicit body representation to synthesize high-quality novel views for arbitrary human performers without per-subject fine-tuning?
• RQ2 How can geometric priors and multi-view source hints be integrated to improve robustness to pose and clothing variation?
• RQ3 Does screen-space occlusion-aware blending enable ghosting-free appearance and better multi-view consistency under sparse inputs?
• RQ4 How does the proposed method perform on unseen subjects and unseen poses compared to state-of-the-art generalizable and case-specific approaches?

Key findings

• GNR achieves leading rendering quality for unseen IDs and unseen poses on GeneBody-1.0 and ZJU-Mocap when compared to generalization baselines.
• GNR demonstrates robust geometry alignment and high-quality appearance even under challenging clothing or poses, outperforming several state-of-the-art methods.
• Ablation studies show that implicit body embedding, attention-based appearance blending, and screen-space occlusion-aware blending each contribute to improvements in geometry, rendering, and appearance fidelity.
• On synthetic RenderPeople data, GNR delivers strong 3D geometry reconstruction (lower Chamfer distances) and high image quality (PSNR/SSIM) relative to baselines.
• The GeneBody-1.0 dataset provides diverse subjects, poses, and clothing to benchmark generalizable human rendering in real-world-like scenarios.
• GNR without body embedding, without attention, or without occlusion-aware blending degrades performance, underscoring the value of each component.