[Paper Review] Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation

Rui Chen, Yongwei Chen+2 more|arXiv (Cornell University)|Mar 24, 2023

Human Motion and Animation14 citations

TL;DR

Fantasia3D decouples geometry and appearance in text-to-3D generation using a DMTet-based hybrid geometry representation and a spatially varying BRDF appearance model, enabling high-quality geometries and photorealistic rendering. It leverages rendered normal maps as shape inputs to a pre-trained diffusion model and learns BRDF parameters for realistic materials.

ABSTRACT

Automatic 3D content creation has achieved rapid progress recently due to the availability of pre-trained, large language models and image diffusion models, forming the emerging topic of text-to-3D content creation. Existing text-to-3D methods commonly use implicit scene representations, which couple the geometry and appearance via volume rendering and are suboptimal in terms of recovering finer geometries and achieving photorealistic rendering; consequently, they are less effective for generating high-quality 3D assets. In this work, we propose a new method of Fantasia3D for high-quality text-to-3D content creation. Key to Fantasia3D is the disentangled modeling and learning of geometry and appearance. For geometry learning, we rely on a hybrid scene representation, and propose to encode surface normal extracted from the representation as the input of the image diffusion model. For appearance modeling, we introduce the spatially varying bidirectional reflectance distribution function (BRDF) into the text-to-3D task, and learn the surface material for photorealistic rendering of the generated surface. Our disentangled framework is more compatible with popular graphics engines, supporting relighting, editing, and physical simulation of the generated 3D assets. We conduct thorough experiments that show the advantages of our method over existing ones under different text-to-3D task settings. Project page and source codes: https://fantasia3d.github.io/.

Motivation & Objective

• Motivate automatic 3D asset creation from text prompts with improved surface quality and materials.
• Decouple geometry and appearance learning to better recover fine geometry and photorealistic textures.
• Leverage a hybrid surface representation (DMTet) to enable explicit surface deformation and differentiable rendering.
• Introduce a spatially varying BRDF model to learn realistic surface materials.
• Ensure compatibility with graphics engines for relighting, editing, and physical simulation.

Proposed method

• Use DMTet as a hybrid geometry representation with a deformable tetrahedral grid and differentiable mesh extraction.
• Render and encode the surface normal map (and object mask early on) as the shape input to a pre-trained image diffusion model via SDS loss.
• Introduce a BRDF-based appearance model learned by an MLP that outputs diffuse, roughness/metalness, and normal variation terms for physically-based rendering.
• Train geometry and appearance models via Score Distillation Sampling (SDS) using the pre-trained Stable Diffusion model.
• Initialize geometry from a 3D ellipsoid or a user-provided shape and iteratively refine through coarse-to-fine stages with geometry and texture optimizations.
• Provide a texture mapping pipeline with UV edge padding to reduce seams and improve render realism.

Experimental results

Research questions

• RQ1 Can disentangled geometry-appearance learning improve the quality of text-to-3D assets compared with entangled or NeRF-based approaches?
• RQ2 Does incorporating a spatially varying BRDF enable photorealistic rendering and better material fidelity for generated surfaces?
• RQ3 Can a normal-map-based shape encoding input to a diffusion model yield finer geometric recovery than color-based encodings?
• RQ4 Is the approach compatible with editing, relighting, and physical simulation in standard graphics engines?

Key findings

• Fantasia3D outperforms existing methods in both geometry quality and appearance realism under zero-shot and user-guided settings.
• Disentangled geometry-appearance learning with DMTet enables fine surface recovery and photorealistic rendering via BRDF materials.
• Using rendered normal maps as shape input to diffusion guidance improves geometry quality over color-based inputs.
• BRDF-based appearance modeling yields more realistic lighting and reflectance than diffuse-only alternatives.
• The method supports relighting, editing, and physical simulation in standard graphics engines like Blender.
• Geometry can be initialized from user-provided shapes or ellipsoids, enabling flexible user-guided generation.