[Paper Review] AvatarCLIP: Zero-Shot Text-Driven Generation and Animation of 3D Avatars

Fangzhou Hong, Mingyuan Zhang+4 more|arXiv (Cornell University)|May 17, 2022

Human Motion and Animation18 citations

TL;DR

AvatarCLIP introduces a zero-shot, text-driven framework for generating and animating 3D avatars using only natural language descriptions. By leveraging CLIP for vision-language supervision and combining shape/motion VAEs with neural rendering, it enables high-fidelity, animatable 3D avatars without expert input or paired training data, achieving strong zero-shot generalization across diverse text prompts.

ABSTRACT

3D avatar creation plays a crucial role in the digital age. However, the whole production process is prohibitively time-consuming and labor-intensive. To democratize this technology to a larger audience, we propose AvatarCLIP, a zero-shot text-driven framework for 3D avatar generation and animation. Unlike professional software that requires expert knowledge, AvatarCLIP empowers layman users to customize a 3D avatar with the desired shape and texture, and drive the avatar with the described motions using solely natural languages. Our key insight is to take advantage of the powerful vision-language model CLIP for supervising neural human generation, in terms of 3D geometry, texture and animation. Specifically, driven by natural language descriptions, we initialize 3D human geometry generation with a shape VAE network. Based on the generated 3D human shapes, a volume rendering model is utilized to further facilitate geometry sculpting and texture generation. Moreover, by leveraging the priors learned in the motion VAE, a CLIP-guided reference-based motion synthesis method is proposed for the animation of the generated 3D avatar. Extensive qualitative and quantitative experiments validate the effectiveness and generalizability of AvatarCLIP on a wide range of avatars. Remarkably, AvatarCLIP can generate unseen 3D avatars with novel animations, achieving superior zero-shot capability.

Motivation & Objective

• To democratize 3D avatar creation by eliminating the need for professional software and expert knowledge.
• To address the lack of high-quality, text-conditioned 3D avatar datasets for supervised training.
• To enable zero-shot generation of novel 3D avatars and motions using only natural language.
• To ensure generated avatars are geometry-accurate, textured, and animatable via neural rendering and motion priors.
• To overcome limitations of existing methods that rely on paired data or suffer from poor generalization.

Proposed method

• Uses a shape VAE to initialize 3D human geometry from text, guided by CLIP embeddings.
• Applies CLIP supervision on rendered images to optimize texture and geometry through neural rendering.
• Imposes template mesh constraints during optimization to preserve plausible human shapes.
• Employs a motion VAE with CLIP-guided reference-based synthesis to generate motion sequences from text.
• Generates candidate poses via latent space optimization and refines them using CLIP-based pose consistency loss.
• Extracts explicit meshes from implicit representations for downstream animation and rendering.

Experimental results

Research questions

• RQ1 Can a zero-shot, text-driven method generate diverse, high-quality 3D avatars without paired training data or expert input?
• RQ2 How well can CLIP supervision guide 3D geometry and texture generation when applied to rendered images?
• RQ3 Can motion VAE priors combined with CLIP enable realistic, text-aligned motion synthesis without motion-text paired data?
• RQ4 How does the method generalize to unseen avatars and complex motion descriptions?
• RQ5 What are the limitations in generating diverse results from the same prompt due to CLIP’s fixed text encoding?

Key findings

• AvatarCLIP achieves superior zero-shot generalization, generating novel 3D avatars with diverse shapes, textures, and motions from natural language prompts.
• The method produces high-quality geometry and textures, as validated by qualitative comparisons showing clear advantages over Text2Mesh and Dream Field.
• Motion synthesis outperforms both direct optimization and sampling from multi-modality Real NVP, with higher scores in user studies.
• Multiple runs with different random seeds yield consistent results, demonstrating robustness compared to unstable baselines like Text2Mesh.
• The framework successfully handles diverse inputs, including celebrity names, fictional characters, and detailed descriptions of clothing and appearance.
• Failure cases occur with complex motions and fine-grained control, indicating limitations in generating out-of-distribution or highly stylized actions.