[Paper Review] HoloDiffusion: Training a 3D Diffusion Model using 2D Images

Animesh Karnewar, Andrea Vedaldi+2 more|arXiv (Cornell University)|Mar 29, 2023

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

HoloDiffusion introduces the first 3D-aware generative diffusion model trained exclusively on posed 2D images, using a hybrid explicit-implicit feature grid and a differentiable rendering loss to enable end-to-end training without 3D supervision. It achieves state-of-the-art 3D view consistency and sample quality on the CO3Dv2 dataset, outperforming existing methods in FID and KID metrics while maintaining scalable, memory-decoupled inference.

ABSTRACT

Diffusion models have emerged as the best approach for generative modeling of 2D images. Part of their success is due to the possibility of training them on millions if not billions of images with a stable learning objective. However, extending these models to 3D remains difficult for two reasons. First, finding a large quantity of 3D training data is much more complex than for 2D images. Second, while it is conceptually trivial to extend the models to operate on 3D rather than 2D grids, the associated cubic growth in memory and compute complexity makes this infeasible. We address the first challenge by introducing a new diffusion setup that can be trained, end-to-end, with only posed 2D images for supervision; and the second challenge by proposing an image formation model that decouples model memory from spatial memory. We evaluate our method on real-world data, using the CO3D dataset which has not been used to train 3D generative models before. We show that our diffusion models are scalable, train robustly, and are competitive in terms of sample quality and fidelity to existing approaches for 3D generative modeling.

Motivation & Objective

• To enable training of 3D diffusion models using only posed 2D images, avoiding the need for large-scale 3D datasets.
• To address the cubic memory and compute complexity of 3D diffusion by decoupling feature grid resolution from rendering resolution.
• To ensure 3D-consistent generation across viewpoints using a learnable, differentiable rendering module.
• To achieve competitive sample quality and fidelity compared to existing 3D generative models, despite training with only 2D supervision.
• To demonstrate scalability and robustness on real-world, category-specific 3D data from the CO3Dv2 dataset.

Proposed method

• The method uses a hybrid explicit-implicit feature grid that stores 3D features and enables differentiable rendering to any viewpoint.
• A pretrained image encoder extracts features from input 2D images, which are then used to condition a 3D feature grid.
• Noise is added to the 3D feature grid, and a 3D U-Net denoiser is trained to reverse this process using a photometric loss between rendered images and ground-truth training images.
• The training objective is defined as the L2 loss between rendered images from the denoised grid and the corresponding real images, enabling end-to-end optimization.
• The feature grid resolution is decoupled from the rendering resolution, reducing memory complexity and enabling high-fidelity rendering without increasing model size.
• The system is trained end-to-end using only posed image supervision, with no access to 3D ground-truth during training.

Experimental results

Research questions

• RQ1 Can a 3D diffusion model be trained effectively using only posed 2D images, without 3D supervision?
• RQ2 How can 3D view consistency be enforced in a diffusion model trained solely on 2D image data?
• RQ3 Can a memory-efficient 3D representation be designed that decouples feature grid resolution from rendering resolution?
• RQ4 How does the performance of a 2D-supervised 3D diffusion model compare to existing 3D generative models in terms of sample quality and fidelity?
• RQ5 Can the model generalize across categories and produce diverse, high-quality 3D-consistent samples from few-view inputs?

Key findings

• HoloDiffusion achieves state-of-the-art FID and KID scores on the CO3Dv2 dataset, with a mean FID of 104.9 and mean KID of 0.091, outperforming EG3D and GET3D.
• On the teddy bear category, HoloDiffusion achieves a FID of 109.2 and KID of 0.106, significantly better than pi-GAN’s 125.8 FID and 0.118 KID.
• The method produces highly view-consistent samples, as demonstrated by qualitative comparisons showing consistent appearance across viewpoints, unlike pi-GAN which lacks 3D consistency.
• HoloDiffusion generalizes well to few-image settings and maintains robust training even with limited supervision.
• The model’s 3D-aware generation is evident in the iterative sampling process, where rendered views evolve consistently from noise to coherent 3D structures.
• The ablation study confirms that the bootstrap version of HoloDiffusion improves performance, indicating the benefit of self-supervised refinement.