[Paper Review] DreamFusion: Text-to-3D using 2D Diffusion

Ben Poole, Ajay Jain+2 more|arXiv (Cornell University)|Sep 29, 2022

Computer Graphics and Visualization Techniques460 citations

TL;DR

DreamFusion uses a pretrained 2D diffusion model as a prior to optimize a NeRF-based 3D scene from text, enabling viewable 3D objects without 3D training data.

ABSTRACT

Recent breakthroughs in text-to-image synthesis have been driven by diffusion models trained on billions of image-text pairs. Adapting this approach to 3D synthesis would require large-scale datasets of labeled 3D data and efficient architectures for denoising 3D data, neither of which currently exist. In this work, we circumvent these limitations by using a pretrained 2D text-to-image diffusion model to perform text-to-3D synthesis. We introduce a loss based on probability density distillation that enables the use of a 2D diffusion model as a prior for optimization of a parametric image generator. Using this loss in a DeepDream-like procedure, we optimize a randomly-initialized 3D model (a Neural Radiance Field, or NeRF) via gradient descent such that its 2D renderings from random angles achieve a low loss. The resulting 3D model of the given text can be viewed from any angle, relit by arbitrary illumination, or composited into any 3D environment. Our approach requires no 3D training data and no modifications to the image diffusion model, demonstrating the effectiveness of pretrained image diffusion models as priors.

Motivation & Objective

• Demonstrate that a pretrained 2D text-to-image diffusion model can serve as a prior for 3D synthesis without 3D data.
• Develop a differentiable image parameterization and a loss that aligns 3D renderings with the diffusion model’s priors.
• Produce coherent 3D objects and scenes that are viewable from arbitrary angles and lighting.
• Show that the approach generalizes to diverse prompts and compares favorably to prior zero-shot text-to-3D methods.

Proposed method

• Use a pretrained 2D diffusion model (Imagen base 64x64) as a fixed prior.
• Adopt a differentiable image parameterization in the form of a NeRF (mip-NeRF 360) to represent 3D geometry and appearance.
• Render NeRF from random camera poses and apply a view-dependent shading model with a random light source.
• Compute gradients via Score Distillation Sampling (SDS), using the diffusion model’s predicted noise to steer NeRF optimization without backpropagating through the diffusion model.
• Optimize NeRF parameters with gradient descent for about 15,000 iterations on TPUv4, using a large classifier-free guidance weight (ω ≈ 100).
• Append view-dependent prompts and random lighting to improve geometry and rendering fidelity.

Experimental results

Research questions

• RQ1 Can a frozen 2D text-to-image diffusion model provide a viable prior for zero-shot text-to-3D synthesis without 3D training data?
• RQ2 How can we optimize a 3D NeRF to produce coherent renderings from multiple random views that align with a diffusion model’s priors?
• RQ3 What role do viewing variations, illumination, and textureless shading play in improving 3D geometry quality under diffusion-guided optimization?
• RQ4 How does DreamFusion’s performance compare with prior zero-shot 3D methods using image-text priors?

Key findings

• DreamFusion can generate coherent 3D NeRF scenes from natural language prompts using only a pretrained 2D diffusion model as a prior.
• Score Distillation Sampling (SDS) provides a simple, differentiable loss enabling optimization in the NeRF parameter space without backpropagating through the diffusion model.
• Randomized camera and lighting sampling across iterations yields view-consistent 3D geometry and enables relighting and environment integration.
• Textureless shading and view-dependent prompts significantly improve geometric fidelity and CLIP-based coherence with prompts.
• Quantitative evaluation using CLIP R-Precision shows DreamFusion outperforming some baselines on color renders and approaching ground-truth captions for certain prompts, while geometry-focused metrics indicate improvements when including shading and textureless geometry.