[Paper Review] Generative AI meets 3D: A Survey on Text-to-3D in AIGC Era

Chenghao Li, Chaoning Zhang+7 more|arXiv (Cornell University)|May 10, 2023

Human Motion and Animation24 citations

TL;DR

A comprehensive survey of text-to-3D methods in the AIGC era, covering 3D data representations, foundation technologies, CLIP and diffusion priors, and 3D applications and challenges.

ABSTRACT

Generative AI has made significant progress in recent years, with text-guided content generation being the most practical as it facilitates interaction between human instructions and AI-generated content (AIGC). Thanks to advancements in text-to-image and 3D modeling technologies, like neural radiance field (NeRF), text-to-3D has emerged as a nascent yet highly active research field. Our work conducts a comprehensive survey on this topic and follows up on subsequent research progress in the overall field, aiming to help readers interested in this direction quickly catch up with its rapid development. First, we introduce 3D data representations, including both Structured and non-Structured data. Building on this pre-requisite, we introduce various core technologies to achieve satisfactory text-to-3D results. Additionally, we present mainstream baselines and research directions in recent text-to-3D technology, including fidelity, efficiency, consistency, controllability, diversity, and applicability. Furthermore, we summarize the usage of text-to-3D technology in various applications, including avatar generation, texture generation, scene generation and 3D editing. Finally, we discuss the agenda for the future development of text-to-3D.

Motivation & Objective

• Explain 3D data representations (Euclidean and non-Euclidean) and their relevance for text-to-3D.
• Review foundation technologies enabling text-to-3D, including NeRF, CLIP, and diffusion models.
• Summarize prominent text-to-3D models and how they leverage image-text priors for 3D generation.
• Survey text-to-3D applications (avatars, textures, scenes, shape transformation) and practical challenges.
• Discuss fidelity, speed, consistency, controllability, and applicability issues in text-to-3D.”

Proposed method

• Classify 3D data representations into Euclidean (voxel grids, multi-view images) and non-Euclidean (meshes, point clouds, neural fields).
• Describe NeRF-based scene representation and differentiable rendering for 3D synthesis.
• Explain CLIP-based text-image priors and their role in guiding 3D generation.
• Summarize diffusion-model-based priors and how they are integrated with NeRFs or explicit 3D representations.
• Review pioneering and follow-up text-to-3D methods (DreamFusion, Magic3D, CLIP-NeRF, CLIP-Mesh, DreamFields, 3D-CLFusion, CLIP-Sculptor, etc.).
• Outline typical pipelines from text prompts to optimized 3D outputs via priors and optimization or latent diffusion.”

Figure 1. Voxel representation of Stanford bunny, the picture is obtained from (Shi et al . , 2022 ) .

Experimental results

Research questions

• RQ1 What 3D data representations are most effective or practical for text-to-3D tasks?
• RQ2 How do NeRF, CLIP, and diffusion priors contribute to text-guided 3D generation?
• RQ3 What are the main architectural and data challenges in text-to-3D, and how do recent works address data scarcity?
• RQ4 What are the primary applications and limitations of text-to-3D in avatars, textures, scenes, and transformations?
• RQ5 What fidelity, speed, consistency, and controllability issues arise in current text-to-3D methods and how might they be mitigated?

Key findings

• Text-to-3D relies on a mix of Euclidean and non-Euclidean representations, with neural fields (e.g., NeRF, SDF) enabling flexible, high-resolution outputs.
• Pre-trained text-to-image diffusion models and CLIP provide strong priors that enable text-guided 3D synthesis with limited 3D data.
• Pioneering methods (DreamFusion, Magic3D, CLIP-NeRF, DreamFields) demonstrate the viability of optimizing 3D content from text prompts using image-text priors, often through NeRF-based representations.
• Follow-up works focus on speeding up generation, improving 3D consistency, and enabling multi-class and editable 3D content (e.g., 3D-CLFusion, CLIP-Sculptor, 3DFuse, CompoNeRF).
• Applications span text-guided 3D avatar, texture, and scene generation, as well as shape transformation, with ongoing challenges in fidelity, speed, and semantic consistency.

Figure 2. Multi-view representation of Stanford bunny, the picture is obtained from (Park et al . , 2016 ) .