[Paper Review] SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes

Xu Chen, Yufeng Zheng+3 more|arXiv (Cornell University)|Jan 1, 2021

3D Shape Modeling and AnalysisEngineering59 references13 citations

TL;DR

SNARF introduces a differentiable forward skinning framework for animating neural implicit 3D shapes, learning a pose-independent skinning weight field and canonical shape from posed meshes without supervision. By using iterative root finding and implicit differentiation, it enables generalization to unseen poses with high-fidelity, pose-dependent deformations, outperforming state-of-the-art methods in reconstruction accuracy and robustness, especially under distribution shift.

ABSTRACT

Neural implicit surface representations have emerged as a promising paradigm to capture 3D shapes in a continuous and resolution-independent manner. However, adapting them to articulated shapes is non-trivial. Existing approaches learn a backward warp field that maps deformed to canonical points. However, this is problematic since the backward warp field is pose dependent and thus requires large amounts of data to learn. To address this, we introduce SNARF, which combines the advantages of linear blend skinning (LBS) for polygonal meshes with those of neural implicit surfaces by learning a forward deformation field without direct supervision. This deformation field is defined in canonical, pose-independent space, allowing for generalization to unseen poses. Learning the deformation field from posed meshes alone is challenging since the correspondences of deformed points are defined implicitly and may not be unique under changes of topology. We propose a forward skinning model that finds all canonical correspondences of any deformed point using iterative root finding. We derive analytical gradients via implicit differentiation, enabling end-to-end training from 3D meshes with bone transformations. Compared to state-of-the-art neural implicit representations, our approach generalizes better to unseen poses while preserving accuracy. We demonstrate our method in challenging scenarios on (clothed) 3D humans in diverse and unseen poses.

Motivation & Objective

• To address the limitation of backward deformation fields in neural implicit representations, which are pose-dependent and generalize poorly to unseen poses.
• To learn a canonical, pose-independent skinning weight field and neural implicit shape from 3D posed meshes without requiring pre-defined skinning weights or part correspondences.
• To enable high-fidelity, continuous, and topology-flexible animation of articulated 3D shapes, including clothed humans, in arbitrary poses.
• To achieve end-to-end training via differentiable forward skinning with analytical gradients derived through implicit differentiation.

Proposed method

• Proposes a forward skinning model that maps deformed space points back to canonical space using iterative root finding to resolve multiple potential canonical correspondences.
• Learns a dense, continuous skinning weight field in canonical space, which is pose-independent and generalizes to unseen poses.
• Uses implicit differentiation to derive analytical gradients of the forward skinning module, enabling end-to-end backpropagation from 3D mesh observations.
• Conditions the neural implicit shape function on pose embeddings to capture non-linear, pose-dependent deformations such as clothing wrinkles and soft tissue movement.
• Trains jointly on sequences of watertight 3D meshes and corresponding bone transformations, without requiring ground-truth skinning weights or part labels.
• Employs a differentiable rendering and optimization pipeline that backpropagates through the forward skinning and neural implicit function to optimize both shape and skinning weights.

Experimental results

Research questions

• RQ1 Can a differentiable forward skinning model be trained end-to-end from 3D posed meshes alone, without supervision on skinning weights or correspondences?
• RQ2 How can multiple canonical correspondences for a deformed point be consistently found in the presence of topology changes, enabling robust deformation learning?
• RQ3 Can a pose-independent skinning field generalize to unseen poses while preserving fine geometric details in neural implicit surfaces?
• RQ4 How does forward skinning compare to backward deformation fields in terms of generalization and reconstruction quality under distribution shift?
• RQ5 Can neural implicit shapes be effectively animated with realistic, non-linear deformations such as clothing dynamics using only mesh and bone data?

Key findings

• SNARF achieves state-of-the-art performance on minimally clothed human benchmarks, outperforming NASA and other baselines by 4.6% in mean IoU for near-surface points.
• On unseen poses, SNARF maintains high reconstruction quality with a 20.4% improvement in near-surface IoU over NASA, demonstrating graceful degradation.
• The method produces realistic, continuous shapes with fine geometric details such as wrinkles and joint deformations, even in extreme, out-of-distribution poses.
• Qualitative results show that SNARF generates plausible shapes with correct body poses and natural clothing deformation, while baselines exhibit artifacts like discontinuities, missing parts, or distorted bodies.
• Learned skinning weights exhibit smooth transitions across body parts, reflecting accurate anatomical assignments without supervision.
• The canonical shape learns pose-dependent correctives, as visualized by heatmaps showing deformation differences from the canonical pose, enabling non-linear shape adaptation.