[Paper Review] HyperNeRFGAN: Hypernetwork approach to 3D NeRF GAN

Adam Kania, Artur Kasymov+5 more|arXiv (Cornell University)|Jan 27, 2023

Computer Graphics and Visualization Techniques4 citations

TL;DR

HyperNeRFGAN proposes a novel GAN architecture that uses a hypernetwork to generate 3D-aware Neural Radiance Fields (NeRF) from 2D images, directly producing NeRF weights from random noise. The model achieves state-of-the-art FID scores on CARLA and ShapeNet datasets, demonstrating superior 3D consistency and image quality over SIREN-based GANs and prior NeRF-GAN methods.

ABSTRACT

The recent surge in popularity of deep generative models for 3D objects has highlighted the need for more efficient training methods, particularly given the difficulties associated with training with conventional 3D representations, such as voxels or point clouds. Neural Radiance Fields (NeRFs), which provide the current benchmark in terms of quality for the generation of novel views of complex 3D scenes from a limited set of 2D images, represent a promising solution to this challenge. However, the training of these models requires the knowledge of the respective camera positions from which the images were viewed. In this paper, we overcome this limitation by introducing HyperNeRFGAN, a Generative Adversarial Network (GAN) architecture employing a hypernetwork paradigm to transform a Gaussian noise into the weights of a NeRF architecture that does not utilize viewing directions in its training phase. Consequently, as evidenced by the findings of our experimental study, the proposed model, despite its notable simplicity in comparison to existing state-of-the-art alternatives, demonstrates superior performance on a diverse range of image datasets where camera position estimation is challenging, particularly in the context of medical data.

Motivation & Objective

• To develop a GAN-based generative model that produces high-quality 3D NeRF representations from unlabeled 2D images.
• To overcome the limitations of existing GANs that rely on SIREN instead of NeRF due to conditioning difficulties in NeRF.
• To enable 3D-aware image synthesis by enforcing 3D consistency through NeRF’s volume rendering and ray tracing.
• To eliminate shared global parameters in the rendering component, allowing dedicated NeRFs per generated object.
• To achieve better 3D generalization and image quality than prior SIREN-based GANs and NeRF-GAN baselines.

Proposed method

• A hypernetwork takes random Gaussian noise as input and generates the full set of weights for a target NeRF model.
• The generated NeRF weights are used to render novel 2D views via volume rendering and ray tracing.
• A standard 2D StyleGAN2 discriminator is used to train the entire GAN architecture in an implicit, end-to-end manner.
• The model is trained exclusively on unlabeled 2D images, with no access to ground-truth 3D geometry.
• The architecture enforces 3D consistency by leveraging NeRF’s differentiable rendering process, which depends on 3D spatial and directional properties.
• The method avoids parameter sharing in the rendering head, ensuring each generated object has a unique, dedicated NeRF representation.

Figure 1: HyperNeRFGAN architecture leverages a hypernetwork paradigm to transfer gaussian noise into weights of NeRF model. After that, we render 2D novel views by NeRF and use a classical 2D discriminator. Our architecture produces 2D images, but we use 3D-aware NeRF representation, which forces t

Experimental results

Research questions

• RQ1 Can a hypernetwork be effectively used to generate NeRF weights for 3D object generation from 2D images?
• RQ2 Does replacing SIREN with NeRF in a GAN framework lead to improved 3D consistency and image quality?
• RQ3 Can a GAN trained on 2D images alone produce 3D-aware outputs with high fidelity and generalization?
• RQ4 How does the proposed HyperNeRFGAN compare quantitatively to SIREN-based GANs like π-GAN and GRAF in 3D generation tasks?
• RQ5 Can the model generate realistic 3D objects with complex material properties, such as transparency in glass, from limited 2D views?

Key findings

• HyperNeRFGAN achieves a Fréchet Inception Distance (FID) of 20.5 on the CARLA dataset, outperforming π-GAN (29.2), GRAF (41.7), and HoloGAN (67.5).
• On the ShapeNet dataset, HyperNeRFGAN achieves FID scores of 29.6 (cars), 33.4 (planes), and 22.0 (chairs), demonstrating strong 3D generation quality.
• On CelebA, HyperNeRFGAN achieves an FID of 15.04 and KID of 0.66, closely matching π-GAN (FID: 14.7, KID: 0.39) while maintaining high inception score (2.63).
• Qualitative results show HyperNeRFGAN successfully models complex material properties such as glass transparency, which π-GAN and GRAF struggle with.
• The model produces high-quality 3D-aware images and can reconstruct NeRFs that are usable for downstream tasks like mesh extraction (e.g., Fig. 6).
• The ablation shows that using NeRF instead of SIREN in the GAN framework leads to better 3D consistency and image quality, validating the core design choice.

Figure 2: Comparison of HyperNeRFGAN and HoloGAN, GRAF, $\pi$ -GAN on CARLA. We obtain similar results to $\pi$ -GAN, but we have a better value of FID score, see Tab 2 .