[Paper Review] DreamWaltz: Make a Scene with Complex 3D Animatable Avatars

Yukun Huang, Jianan Wang+6 more|arXiv (Cornell University)|May 21, 2023

Human Motion and Animation16 citations

TL;DR

DreamWaltz introduces a text-to-avatar framework that creates high-quality, animatable 3D avatars using SMPL priors and 3D-consistent SDS to enable pose-conditioned animation and scene composition without retraining.

ABSTRACT

We present DreamWaltz, a novel framework for generating and animating complex 3D avatars given text guidance and parametric human body prior. While recent methods have shown encouraging results for text-to-3D generation of common objects, creating high-quality and animatable 3D avatars remains challenging. To create high-quality 3D avatars, DreamWaltz proposes 3D-consistent occlusion-aware Score Distillation Sampling (SDS) to optimize implicit neural representations with canonical poses. It provides view-aligned supervision via 3D-aware skeleton conditioning which enables complex avatar generation without artifacts and multiple faces. For animation, our method learns an animatable 3D avatar representation from abundant image priors of diffusion model conditioned on various poses, which could animate complex non-rigged avatars given arbitrary poses without retraining. Extensive evaluations demonstrate that DreamWaltz is an effective and robust approach for creating 3D avatars that can take on complex shapes and appearances as well as novel poses for animation. The proposed framework further enables the creation of complex scenes with diverse compositions, including avatar-avatar, avatar-object and avatar-scene interactions. See https://dreamwaltz3d.github.io/ for more vivid 3D avatar and animation results.

Motivation & Objective

• Motivate the need for easily controllable, high-quality, and animatable 3D avatars from text prompts.
• Leverage human body priors (SMPL) to constrain and guide 3D avatar generation.
• Develop 3D-consistent, occlusion-aware supervision to avoid common diffusion-based artifacts.
• Learn an animatable NeRF representation that can be retargeted to arbitrary poses without retraining.
• Demonstrate scene creation with avatar–avatar, avatar–object, and avatar–scene interactions.

Proposed method

• Use a trainable NeRF as the 3D avatar representation.
• Employ SMPL prior for initialization and to extract 3D-aware skeletons for conditioning diffusion-based SDS.
• Apply 3D-consistent Score Distillation Sampling with occlusion culling to ensure coherent 3D geometry and avoid multi-face artifacts.
• Condition SDS with skeleton-based conditioning images via ControlNet to align views with NeRF rendering.
• Learn an animatable avatar by training a density weighting network that enables pose-driven deformation without retraining.
• Incorporate a human pose prior (VPoser) to generalize the density weighting to arbitrary poses.

Experimental results

Research questions

• RQ1 Can text prompts generate high-quality, complex 3D avatars that are animatable from arbitrary poses?
• RQ2 Does SMPL-guided initialization and 3D-aware SDS reduce artifacts like Janus faces while preserving detail?
• RQ3 Can an animatable NeRF be trained to retarget to any pose sequence without additional retraining?
• RQ4 Is it feasible to compose avatars into complex scenes with diverse interactions (avatar–avatar, avatar–object, avatar–scene) using diffusion-based supervision?

Key findings

• DreamWaltz achieves high-quality 3D avatars with complex shapes and textures across viewpoints.
• Occlusion-aware 3D-consistent SDS reduces view-ambiguity and artifacts such as multi-face issues.
• An animatable NeRF learned with pose-conditioned diffusion supervision can be animated for arbitrary poses without retraining.
• The framework enables scene composition with avatars and other assets and supports avatar–avatar and avatar–scene interactions.
• User studies show DreamWaltz avatars outperform baselines in geometry and texture quality.
• A density weighting network stabilizes articulation by mitigating transformation-induced artifacts in non-skin-tight avatars.