[Paper Review] GRAM-HD: 3D-Consistent Image Generation at High Resolution with Generative Radiance Manifolds

Jianfeng Xiang, Jiaolong Yang+2 more|arXiv (Cornell University)|Jun 15, 2022

Advanced Vision and Imaging4 citations

TL;DR

GRAM-HD proposes a 3D-aware GAN that generates high-resolution (up to 1024×1024) images with strict multiview consistency by performing super-resolution in 3D space via 2D CNNs on learned radiance manifolds. It achieves state-of-the-art results in image quality and 3D consistency while reducing memory and inference costs compared to prior methods.

ABSTRACT

Recent works have shown that 3D-aware GANs trained on unstructured single image collections can generate multiview images of novel instances. The key underpinnings to achieve this are a 3D radiance field generator and a volume rendering process. However, existing methods either cannot generate high-resolution images (e.g., up to 256X256) due to the high computation cost of neural volume rendering, or rely on 2D CNNs for image-space upsampling which jeopardizes the 3D consistency across different views. This paper proposes a novel 3D-aware GAN that can generate high resolution images (up to 1024X1024) while keeping strict 3D consistency as in volume rendering. Our motivation is to achieve super-resolution directly in the 3D space to preserve 3D consistency. We avoid the otherwise prohibitively-expensive computation cost by applying 2D convolutions on a set of 2D radiance manifolds defined in the recent generative radiance manifold (GRAM) approach, and apply dedicated loss functions for effective GAN training at high resolution. Experiments on FFHQ and AFHQv2 datasets show that our method can produce high-quality 3D-consistent results that significantly outperform existing methods. It makes a significant step towards closing the gap between traditional 2D image generation and 3D-consistent free-view generation.

Motivation & Objective

• To enable high-resolution (up to 1024×1024) 3D-consistent image generation from unstructured 2D image collections.
• To overcome the computational bottleneck of neural volume rendering in high-resolution 3D-aware GANs.
• To preserve strict 3D consistency across views while achieving super-resolution, avoiding 2D CNN upsampling in image space.
• To reduce memory and inference cost compared to existing 3D-aware GANs without sacrificing image quality or consistency.

Proposed method

• Leverages the generative radiance manifold (GRAM) framework to define 3D surface manifolds that represent object geometry and radiance.
• Performs super-resolution by flattening and sampling manifolds onto 2D grids, then applying a shared 2D CNN for upsampling and feature-to-radiance translation.
• Introduces a patch-based adversarial loss ($\mathcal{L}_{\mathrm{patch}}$) to suppress checkerboard artifacts in high-resolution outputs.
• Applies a cross-resolution consistency loss ($\mathcal{L}_{\mathrm{cons}}$) to stabilize training and prevent spurious artifacts like floating objects.
• Uses a shared generator network with style modulation and a separate background radiance network to improve realism.
• Employs a multi-scale training strategy that starts from low-resolution manifolds and progressively refines to high-resolution outputs.

Figure 2: The overall framework of our GRAM-HD method. The generator consists of two components: the radiance manifold generator and the manifold super-resolution module. The former generates radiance and feature manifolds that represent an LR 3D scene. Through manifold gridding, the manifolds are s

Experimental results

Research questions

• RQ1 Can 3D-aware GANs generate high-resolution (1024×1024) images while maintaining strict multiview consistency?
• RQ2 Is it possible to achieve 3D super-resolution efficiently without relying on expensive 3D convolutions or 2D image-space upsampling?
• RQ3 How can adversarial training be stabilized at high resolution to avoid artifacts like checkerboarding and spurious objects?
• RQ4 What loss functions are most effective for preserving 3D consistency and image quality during high-resolution refinement?
• RQ5 Can the computational cost of 3D-aware GANs be significantly reduced without compromising generation quality?

Key findings

• GRAM-HD achieves a Fréchet Inception Distance (FID) of 11.8 on FFHQ-256², representing a 21% reduction compared to the baseline GRAM method.
• Inference time for 1024×1024 generation is reduced by 95% compared to prior methods, demonstrating significant efficiency gains.
• Memory usage is reduced by 76% at 256×256 resolution, and inference time is reduced by 58% compared to GRAM.
• The patch adversarial loss ($\mathcal{L}_{\mathrm{patch}}$) effectively eliminates checkerboard artifacts in high-resolution outputs.
• The cross-resolution consistency loss ($\mathcal{L}_{\mathrm{cons}}$) prevents spurious artifacts such as floating objects in the foreground.
• Real-time free-view synthesis at 1024×1024 resolution is achieved at 90 FPS using cached mesh representations.

Figure 3: Left: Rendered images and radiance maps on the surface manifolds. Three sampled manifolds are shown here. The corresponding LR results before super-resolution are presented at bottom right. Right: Extracted proxy 3D shapes at HR (top row) and LR (bottom row). The rendered images are shown