[Paper Review] IPDreamer: Appearance-Controllable 3D Object Generation with Complex Image Prompts

Bohan Zeng, Shanglin Li+10 more|arXiv (Cornell University)|Oct 9, 2023

3D Shape Modeling and Analysis4 citations

TL;DR

IPDreamer introduces a novel text-to-3D generation framework that enables appearance-controllable 3D object synthesis by integrating image prompts into the optimization process via Image Prompt Score Distillation (IPSD). By leveraging image prompt embeddings and normal maps, IPDreamer achieves high-fidelity, style-transfer-capable 3D generation that outperforms SOTA methods in controllability and realism, even when textual and image prompts conflict.

ABSTRACT

Recent advances in 3D generation have been remarkable, with methods such as DreamFusion leveraging large-scale text-to-image diffusion-based models to guide 3D object generation. These methods enable the synthesis of detailed and photorealistic textured objects. However, the appearance of 3D objects produced by such text-to-3D models is often unpredictable, and it is hard for single-image-to-3D methods to deal with images lacking a clear subject, complicating the generation of appearance-controllable 3D objects from complex images. To address these challenges, we present IPDreamer, a novel method that captures intricate appearance features from complex $ extbf{I}$mage $ extbf{P}$rompts and aligns the synthesized 3D object with these extracted features, enabling high-fidelity, appearance-controllable 3D object generation. Our experiments demonstrate that IPDreamer consistently generates high-quality 3D objects that align with both the textual and complex image prompts, highlighting its promising capability in appearance-controlled, complex 3D object generation. Our code is available at https://github.com/zengbohan0217/IPDreamer.

Motivation & Objective

• To address the lack of appearance controllability in existing text-to-3D generation methods, where generated 3D objects exhibit random and uncontrolled textures due to stochastic LoRA fine-tuning.
• To improve upon existing score distillation methods like SDS and VSD by incorporating richer appearance guidance through image prompts rather than relying solely on textual descriptions.
• To enable style transfer from reference images to 3D objects, even when the reference image structure differs significantly from the target 3D geometry.
• To develop a unified framework that simultaneously optimizes 3D mesh geometry and texture using image prompt embeddings and normal maps for enhanced visual fidelity.

Proposed method

• IPDreamer employs Zero-1-to-3 for multiview generation to provide strong 3D prior for NeRF-based 3D representation.
• It uses IP-Adapter to extract image prompt embeddings from both the generated reference images and their corresponding normal maps.
• Image Prompt Score Distillation (IPSD) is introduced as a new optimization objective that guides both texture and geometry optimization using image prompt embeddings.
• The method optimizes 3D meshes by minimizing a combined loss that includes a geometry-aware regularization term δ_geo and a perceptual loss based on image prompt features.
• The framework uses a parametric score model to treat textual prompts as random variables, improving diversity and reducing over-smoothing compared to standard SDS.
• For single-image-to-3D generation, IPDreamer combines IPSD with Zero-123 to preserve identity and detail while enabling style transfer.

Experimental results

Research questions

• RQ1 Can image prompts provide more effective and controllable appearance guidance than textual prompts alone in 3D object generation?
• RQ2 How does incorporating image prompt embeddings and normal maps improve the fidelity and style alignment of generated 3D objects?
• RQ3 To what extent can IPDreamer transfer visual styles from diverse reference images to 3D meshes, even when the image and mesh structures differ?
• RQ4 Does the proposed IPSD loss outperform existing score distillation methods like SDS and VSD in generating high-quality, diverse, and controllable 3D objects?
• RQ5 Can IPDreamer effectively balance identity preservation and style transfer in single-image-to-3D generation?

Key findings

• IPDreamer successfully transfers visual styles from image prompts to 3D objects, even when the reference image and 3D mesh have different structural layouts.
• In cases where textual and image prompts conflict (e.g., 'iron' vs. 'leather' appearance), the image prompt dominates the final 3D output, demonstrating strong appearance control.
• The ablation study confirms that the geometry optimization loss ℒ_IPSD-Geo effectively captures high-frequency details from image prompts, as shown in refined normal maps.
• The introduction of δ_geo significantly improves texture quality and identity preservation, with visual comparisons showing superior results over methods without this term.
• IPDreamer outperforms SOTA methods like DreamFusion, Magic3D, Fantasia3D, and ProlificDreamer in both qualitative and quantitative evaluations, especially in appearance controllability and style alignment.
• The framework enables high-quality 3D generation from both text-and-image prompts and single-image-to-3D generation, with improved detail and consistency compared to baseline methods.