[Paper Review] 3D-aware Image Generation using 2D Diffusion Models

Jianfeng Xiang, Jiaolong Yang+2 more|arXiv (Cornell University)|Mar 31, 2023

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes a novel 3D-aware image generation method that leverages 2D diffusion models by formulating multiview image generation as a sequential unconditional–conditional sampling process. It achieves state-of-the-art performance on ImageNet, generating high-fidelity, 360° consistent views from unaligned, in-the-wild images using monocular depth estimation and data augmentation.

ABSTRACT

In this paper, we introduce a novel 3D-aware image generation method that leverages 2D diffusion models. We formulate the 3D-aware image generation task as multiview 2D image set generation, and further to a sequential unconditional-conditional multiview image generation process. This allows us to utilize 2D diffusion models to boost the generative modeling power of the method. Additionally, we incorporate depth information from monocular depth estimators to construct the training data for the conditional diffusion model using only still images. We train our method on a large-scale dataset, i.e., ImageNet, which is not addressed by previous methods. It produces high-quality images that significantly outperform prior methods. Furthermore, our approach showcases its capability to generate instances with large view angles, even though the training images are diverse and unaligned, gathered from "in-the-wild" real-world environments.

Motivation & Objective

• To enable 3D-aware image generation using powerful 2D diffusion models, which have not been effectively applied to 3D generation tasks.
• To address the challenge of generating consistent multiview images from unstructured, in-the-wild 2D datasets without requiring 3D supervision.
• To overcome the lack of multiview training data by leveraging monocular depth estimation to synthesize multiview supervision from still images.
• To improve generation quality under large view angles (up to 360°) using data augmentation strategies that reduce domain gaps between training and inference.
• To demonstrate the feasibility and effectiveness of 3D-aware generation on large-scale, diverse datasets like ImageNet, which previous methods have not addressed.

Proposed method

• The method formulates 3D-aware image generation as sequential multiview image generation: first sampling an initial view unconditionally, then iteratively generating subsequent views conditioned on previously generated views.
• It uses the chain rule of probability to factorize the joint distribution of multiple views, enabling variable-length output and efficient autoregressive generation.
• Monocular depth estimation is applied to single images to generate depth maps, which are then used to synthesize multiple view projections for training.
• A conditional diffusion model is trained to generate each subsequent view based on the previous views and their corresponding depth maps, enabling consistent 3D geometry across views.
• Data augmentation strategies—specifically blur and texture erosion augmentation—are introduced to reduce domain gaps and improve generalization, especially for large-view-angle generation.
• For high-resolution generation, a diffusion-based super-resolution model is fine-tuned from a pretrained 128² model to upsample 128² outputs to 256² resolution.

Experimental results

Research questions

• RQ1 Can 2D diffusion models be effectively adapted for 3D-aware image generation without requiring explicit 3D supervision or NeRF-based representations?
• RQ2 How can multiview image generation be modeled as a sequential conditional generation process using diffusion models?
• RQ3 Can monocular depth estimation be used to synthesize sufficient multiview training data from unaligned, in-the-wild 2D images to enable 3D-aware generation?
• RQ4 What data augmentation strategies are most effective in reducing domain gaps and improving performance on large-view-angle generation?
• RQ5 Can this method achieve state-of-the-art results on large-scale, diverse datasets like ImageNet, which previous 3D-aware GAN-based methods have not addressed?

Key findings

• The proposed method achieves a Fréchet Inception Distance (FID) of 9.45 on ImageNet, significantly outperforming prior SOTA 3D-aware GANs such as EG3D (FID 40.4) and pi-GAN (FID 138).
• On the SDIP Dog dataset, the method achieves a FID of 18.1 for 9 views with 17° yaw range, improving over ablation variants without erosion (20.9) or blur (22.2) augmentation.
• The method successfully generates consistent 360° view sequences from unaligned training data, demonstrating robustness to diverse and unstructured inputs.
• The fusion-based view synthesis method achieves 16 fps inference speed and maintains low FID (14.1) on ImageNet, showing practical applicability for arbitrary-view generation.
• The high-resolution generation pipeline successfully upsamples 128² outputs to 256² using a fine-tuned diffusion super-resolution model, preserving image quality and 3D consistency.
• Ablation studies confirm that texture erosion and blur augmentations are critical for reducing artifacts and improving performance on large-view-angle synthesis.