[Paper Review] Instruct 3D-to-3D: Text Instruction Guided 3D-to-3D conversion

Hiromichi Kamata, Yuiko Sakuma+3 more|arXiv (Cornell University)|Mar 28, 2023

Advanced Vision and Imaging4 citations

TL;DR

Instruct 3D-to-3D proposes a text instruction-guided 3D-to-3D conversion method that leverages pretrained image-to-image diffusion models conditioned on the source 3D scene for high-fidelity, consistent, and controllable 3D generation. It achieves superior quality and 3D consistency over baselines by integrating source scene conditioning and dynamic scaling for adjustable geometry transformation strength.

ABSTRACT

We propose a high-quality 3D-to-3D conversion method, Instruct 3D-to-3D. Our method is designed for a novel task, which is to convert a given 3D scene to another scene according to text instructions. Instruct 3D-to-3D applies pretrained Image-to-Image diffusion models for 3D-to-3D conversion. This enables the likelihood maximization of each viewpoint image and high-quality 3D generation. In addition, our proposed method explicitly inputs the source 3D scene as a condition, which enhances 3D consistency and controllability of how much of the source 3D scene structure is reflected. We also propose dynamic scaling, which allows the intensity of the geometry transformation to be adjusted. We performed quantitative and qualitative evaluations and showed that our proposed method achieves higher quality 3D-to-3D conversions than baseline methods.

Motivation & Objective

• To address the lack of controllable and high-quality 3D-to-3D editing guided by natural language instructions.
• To improve 3D consistency and structural fidelity during text-guided 3D scene transformation.
• To enable users to control the strength of geometric changes during 3D-to-3D conversion.
• To achieve high-fidelity 3D generation by leveraging pretrained image-to-image diffusion models conditioned on both text and source 3D geometry.
• To provide a method that preserves semantic and structural content of the source scene while enabling expressive edits based on text.

Proposed method

• The method uses a pretrained image-to-image diffusion model to maximize likelihood of rendered 2D views from the target 3D scene, ensuring high image quality.
• The source 3D scene is explicitly used as a conditioning input to preserve structural and semantic consistency during generation.
• Dynamic scaling is introduced to gradually adjust the voxel grid resolution, enabling smooth and controllable geometry transformations.
• The 3D representation is based on a voxel grid-based implicit model, with geometry changes applied through iterative denoising in a diffusion process.
• Classifier-free guidance is applied during the diffusion process to control the influence of the text instruction on the output.
• The method optimizes the NeRF-like 3D representation by minimizing the noise prediction loss between denoised views and target images.

Figure 1: Overview of Instruct 3D-to-3D. First, the target model is initialized with the source model (i). Next, the target image is rendered from a random camera viewpoint (ii) and then the noise is added to input into InstructPix2Pix. The source image is rendered from the same viewpoint (iii) and

Experimental results

Research questions

• RQ1 Can text instruction-guided 3D-to-3D conversion achieve higher image quality and 3D consistency than existing methods?
• RQ2 How can the strength of geometric transformation be controlled during 3D-to-3D editing?
• RQ3 Can conditioning on the source 3D scene improve structural fidelity and reduce distribution shift in generated scenes?
• RQ4 How does dynamic scaling compare to conventional progressive scaling in preserving 3D structure during transformation?
• RQ5 What is the impact of spatial reasoning limitations in text-guided 3D editing, especially for complex instructions?

Key findings

• Instruct 3D-to-3D achieves a lower BRISQUE score (indicating higher image quality) than DreamFusion and CLIP-NeRF, demonstrating superior visual quality.
• The method achieves CLIP scores comparable to DreamFusion, indicating strong text-image alignment, despite not being explicitly optimized for CLIP.
• User preference studies show that Instruct 3D-to-3D is preferred over DreamFusion and CLIP-NeRF in 72% of cases, indicating higher perceptual quality.
• Dynamic scaling preserves 3D structure better than progressive scaling, which causes significant structural damage due to abrupt voxel reduction.
• Larger dynamic scaling factors lead to more significant structural changes, confirming controllability of transformation intensity.
• Failure cases occur with spatial reasoning tasks (e.g., placing an object on a chair), where the model incorrectly alters structural components, indicating a need for depth-aware reasoning.

Figure 2: Comparison of progressive scaling and proposed dynamic scaling. In dynamic scaling, we first gradually decrease the number of voxels to change the global structure, then increase them to change to the local structure.