[Paper Review] InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Prior

Zhiheng Liu, Hao Ouyang+8 more|arXiv (Cornell University)|Apr 17, 2024

3D Shape Modeling and Analysis4 citations

TL;DR

InFusion proposes a novel 3D Gaussian inpainting method that leverages a diffusion prior-based depth completion model to generate high-fidelity, photorealistic 3D scene completions. By using image-conditioned depth inpainting to initialize 3D Gaussians, the approach achieves ~20× faster inference and superior visual quality compared to prior methods, enabling efficient and interactive editing of 3D scenes with user-defined textures or inserted objects.

ABSTRACT

3D Gaussians have recently emerged as an efficient representation for novel view synthesis. This work studies its editability with a particular focus on the inpainting task, which aims to supplement an incomplete set of 3D Gaussians with additional points for visually harmonious rendering. Compared to 2D inpainting, the crux of inpainting 3D Gaussians is to figure out the rendering-relevant properties of the introduced points, whose optimization largely benefits from their initial 3D positions. To this end, we propose to guide the point initialization with an image-conditioned depth completion model, which learns to directly restore the depth map based on the observed image. Such a design allows our model to fill in depth values at an aligned scale with the original depth, and also to harness strong generalizability from largescale diffusion prior. Thanks to the more accurate depth completion, our approach, dubbed InFusion, surpasses existing alternatives with sufficiently better fidelity and efficiency under various complex scenarios. We further demonstrate the effectiveness of InFusion with several practical applications, such as inpainting with user-specific texture or with novel object insertion.

Motivation & Objective

• To address the challenge of high-fidelity 3D Gaussian inpainting in complex scenes with missing geometry and texture.
• To improve the accuracy and efficiency of 3D point initialization by leveraging depth completion from pre-trained diffusion models.
• To enable interactive 3D editing, such as user-defined texture inpainting and novel object insertion, within 3D Gaussian scenes.
• To overcome limitations of prior methods that suffer from blurred textures and slow convergence due to poor initial 3D point placement.

Proposed method

• The method first performs image-conditioned depth inpainting using a pre-trained latent diffusion model (LDM) to generate high-quality depth maps aligned with the original scene geometry.
• The inpainted depth maps are then unprojected into 3D space to generate initial 3D Gaussian points, ensuring geometric consistency with the scene.
• A progressive refinement strategy is employed for complex occlusions, using multiple views to improve depth completion and point initialization accuracy.
• The framework supports interactive editing by allowing users to modify the reference image, which directly influences the texture and appearance of the inpainted 3D region.
• The approach integrates strong generative priors from diffusion models to enhance depth completion, especially for object boundaries and complex structures.
• The method is trained end-to-end to optimize both depth completion and subsequent 3D Gaussian rendering quality, minimizing artifacts and misalignment.

Experimental results

Research questions

• RQ1 Can a diffusion-based depth completion model significantly improve the fidelity and efficiency of 3D Gaussian inpainting compared to conventional methods?
• RQ2 How does using depth completion from a pre-trained diffusion model affect the geometric consistency and visual quality of generated 3D Gaussians?
• RQ3 To what extent can the proposed method support interactive 3D editing tasks such as user-defined texture inpainting and object insertion?
• RQ4 How does progressive multi-view refinement enhance performance in scenes with large occlusions or complex geometry?
• RQ5 Can the integration of diffusion priors enable robust completion of structured objects and backgrounds in both forward-facing and 360-degree scenes?

Key findings

• InFusion achieves a 20× speedup in optimization time compared to baseline methods, reducing inference time from 5 hours to 40 seconds on the SPIn-NeRF dataset.
• The method achieves the lowest LPIPS score (0.421) and FID score (92.62) among all compared methods, indicating superior perceptual and statistical similarity to ground truth.
• Visual comparisons show that InFusion produces sharper textures and better geometric alignment than baselines, especially in multi-object occlusion scenarios.
• The use of diffusion prior in depth completion significantly reduces depth discontinuities and misalignment, leading to more coherent 3D scene extensions.
• The method enables interactive texture editing and object insertion, which are not supported by prior approaches due to their reliance on fixed or coarse initialization.
• Ablation studies confirm that diffusion-based depth inpainting outperforms image-based inpainting and depth alignment techniques, particularly in preserving object structure and depth continuity.