[Paper Review] MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images

Shaofei Wang, Marko Mihajlović+3 more|arXiv (Cornell University)|Jun 22, 2021

3D Shape Modeling and AnalysisEngineering71 references48 citations

TL;DR

MetaAvatar learns a meta-learned hypernetwork that predicts dynamic neural SDFs for clothed humans from few monocular depth images, enabling fast, controllable avatars with pose-dependent cloth deformations.

ABSTRACT

In this paper, we aim to create generalizable and controllable neural signed distance fields (SDFs) that represent clothed humans from monocular depth observations. Recent advances in deep learning, especially neural implicit representations, have enabled human shape reconstruction and controllable avatar generation from different sensor inputs. However, to generate realistic cloth deformations from novel input poses, watertight meshes or dense full-body scans are usually needed as inputs. Furthermore, due to the difficulty of effectively modeling pose-dependent cloth deformations for diverse body shapes and cloth types, existing approaches resort to per-subject/cloth-type optimization from scratch, which is computationally expensive. In contrast, we propose an approach that can quickly generate realistic clothed human avatars, represented as controllable neural SDFs, given only monocular depth images. We achieve this by using meta-learning to learn an initialization of a hypernetwork that predicts the parameters of neural SDFs. The hypernetwork is conditioned on human poses and represents a clothed neural avatar that deforms non-rigidly according to the input poses. Meanwhile, it is meta-learned to effectively incorporate priors of diverse body shapes and cloth types and thus can be much faster to fine-tune, compared to models trained from scratch. We qualitatively and quantitatively show that our approach outperforms state-of-the-art approaches that require complete meshes as inputs while our approach requires only depth frames as inputs and runs orders of magnitudes faster. Furthermore, we demonstrate that our meta-learned hypernetwork is very robust, being the first to generate avatars with realistic dynamic cloth deformations given as few as 8 monocular depth frames.

Motivation & Objective

• Motivate generalizable clothed human modeling that supports pose-dependent cloth deformations.
• Develop a meta-learning framework to initialize a hypernetwork that yields dynamic neural SDFs conditioned on body poses.
• Enable fast fine-tuning from monocular depth images to create subject-specific animatable avatars.
• Compare against mesh-input baselines and demonstrate robustness with very limited data.

Proposed method

• Use a meta-learned hypernetwork g_ψ to predict neural SDF parameters for clothed humans in canonical space.
• Meta-learn a static neural SDF f_φ via few-shot meta-learning (IGR loss) to capture priors across body shapes and clothing.
• Train a hypernetwork to predict residuals to the static meta-SDF parameters when conditioned on bone transformations {B_b}.
• Canonicalize input depth points using inverse skinning network; then animate via forward skinning network.
• Optimize the hypernetwork during test-time with a small fine-tuning set of depth frames and SMPL fittings.
• Encode bone transformations with a hierarchical MLP encoder for g_ψ.

Experimental results

Research questions

• RQ1 Can a meta-learned prior enable fast, few-shot generation of animatable clothed human avatars from depth images?
• RQ2 How well does a hypernetwork-based dynamic SDF model generalize across subjects, clothing types, and poses with limited input data?
• RQ3 What are the benefits of meta-learning a static SDF prior versus meta-learning a dynamic hypernetwork for pose-dependent cloth deformations?
• RQ4 How does MetaAvatar perform relative to mesh-based or full-input baselines when only depth frames are available?

Key findings

• MetaAvatar produces controllable dynamic neural SDF avatars from as few as 8 depth frames in about 2 minutes of fine-tuning.
• The method outperforms baselines that require complete meshes (NASA, SCANimate) and approaches LEAP while providing better cloth detail under depth-only input.
• A two-stage meta-learning strategy (static meta-SDF initialization, then hypernetwork residuals) stabilizes learning and enables fast adaptation.
• The learned deformation prior is robust to unseen subjects, poses, and at least one unseen cloth type in experiments on CAPE.
• With reduced fine-tuning data (down to <1%), MetaAvatar maintains competitive perceptual quality and reasonable interpolation/extrapolation performance.