[Paper Review] Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control

Lingjie Liu, Marc Habermann+4 more|arXiv (Cornell University)|Jun 3, 2021

3D Shape Modeling and AnalysisEngineering65 references58 citations

TL;DR

Neural Actor (NA) enables high-quality free-view synthesis of moving human actors with controllable novel poses by deforming a neural radiance field into a canonical pose space guided by SMPL, and by using texture maps as latent cues to model pose-dependent appearance.

ABSTRACT

We propose Neural Actor (NA), a new method for high-quality synthesis of humans from arbitrary viewpoints and under arbitrary controllable poses. Our method is built upon recent neural scene representation and rendering works which learn representations of geometry and appearance from only 2D images. While existing works demonstrated compelling rendering of static scenes and playback of dynamic scenes, photo-realistic reconstruction and rendering of humans with neural implicit methods, in particular under user-controlled novel poses, is still difficult. To address this problem, we utilize a coarse body model as the proxy to unwarp the surrounding 3D space into a canonical pose. A neural radiance field learns pose-dependent geometric deformations and pose- and view-dependent appearance effects in the canonical space from multi-view video input. To synthesize novel views of high fidelity dynamic geometry and appearance, we leverage 2D texture maps defined on the body model as latent variables for predicting residual deformations and the dynamic appearance. Experiments demonstrate that our method achieves better quality than the state-of-the-arts on playback as well as novel pose synthesis, and can even generalize well to new poses that starkly differ from the training poses. Furthermore, our method also supports body shape control of the synthesized results.

Motivation & Objective

• Motivate high-quality free-view synthesis of humans from arbitrary viewpoints and poses in everyday attire.
• Develop a geometry-guided deformable NeRF framework that uses SMPL as a canonical-space proxy to handle large pose-induced deformations.
• Address ambiguity in dynamic geometry and appearance by incorporating 2D texture maps on the SMPL surface as latent variables and adversarial texture prior.
• Enable pose-controlled playback and novel-pose synthesis, plus shape manipulation of the actor without re-scanning.
• Provide a new multi-view, dense-camera human performance dataset and demonstrate improved rendering quality over prior methods.

Proposed method

• Use SMPL as a canonical-space proxy to deform points via inverse skinning and learn a pose-dependent residual deformation.
• Represent the radiance field in the canonical space with a NeRF-like network predicting color and density from deformed points.
• Predict pose-dependent texture maps Z on the SMPL UV map to capture local appearance and dynamic effects, with an auxiliary texture-feature extractor G(Z).
• Train a texture prior via an image-to-image translation network (normal maps to texture maps) to produce frame-consistent textures, using adversarial loss to avoid regressions to mean appearance.
• Render via geometry-guided ray marching by sampling along rays and querying the deformable radiance field in the canonical space, with texture features conditioning color predictions.
• Incorporate an adversarial loss on texture maps to mitigate uncertainty and improve realism of pose-dependent details.

Experimental results

Research questions

• RQ1 Can a geometry-guided deformable NeRF, guided by an SMPL model, synthesize high-fidelity novel views of freely moving humans under novel poses?
• RQ2 Does conditioning NeRF on pose through inverse-skinned canonical-space deformations improve handling of large body motions compared to pose-augmented NeRFs?
• RQ3 Can 2D texture maps defined on the SMPL surface serve as effective latent variables to model pose-dependent geometry and appearance without blurring?
• RQ4 Is it possible to generalize to unseen poses and support body-shape variations while maintaining photorealism?
• RQ5 Can the method enable practical shape manipulation (e.g., resizing actors) without extra scanning or rigging?

Key findings

• NA achieves higher rendering quality than state-of-the-art baselines on playback and novel-pose synthesis tasks.
• The SMPL-guided deformation enables efficient handling of large movements by decoupling rigid skinning from non-rigid residual deformation.
• Texture maps on the SMPL surface as latent variables, together with an adversarial texture prior, reduce blurriness and capture pose-dependent local details.
• The approach generalizes to novel poses that differ significantly from training poses and supports reshaping the actor’s body for free-view synthesis.
• A new multi-view dense-camera dataset was captured and used to validate performance, with publicly available data promised.