[Paper Review] CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesis

Peng Zhou, Lingxi Xie+2 more|arXiv (Cornell University)|Oct 19, 2021

Generative Adversarial Networks and Image SynthesisComputer Science56 references74 citations

TL;DR

CIPS-3D introduces a style-based, 3D-aware generator that renders each pixel independently using a shallow NeRF for 3D shape and a deep INR for appearance, achieving state-of-the-art 3D-aware image quality on FFHQ.

ABSTRACT

The style-based GAN (StyleGAN) architecture achieved state-of-the-art results for generating high-quality images, but it lacks explicit and precise control over camera poses. The recently proposed NeRF-based GANs made great progress towards 3D-aware generators, but they are unable to generate high-quality images yet. This paper presents CIPS-3D, a style-based, 3D-aware generator that is composed of a shallow NeRF network and a deep implicit neural representation (INR) network. The generator synthesizes each pixel value independently without any spatial convolution or upsampling operation. In addition, we diagnose the problem of mirror symmetry that implies a suboptimal solution and solve it by introducing an auxiliary discriminator. Trained on raw, single-view images, CIPS-3D sets new records for 3D-aware image synthesis with an impressive FID of 6.97 for images at the $256 imes256$ resolution on FFHQ. We also demonstrate several interesting directions for CIPS-3D such as transfer learning and 3D-aware face stylization. The synthesis results are best viewed as videos, so we recommend the readers to check our github project at https://github.com/PeterouZh/CIPS-3D

Motivation & Objective

• Motivate explicit control over camera pose in high-fidelity 3D-aware image synthesis.
• Propose a generator that avoids upsampling/convolutions by synthesizing each pixel independently.
• Balance memory efficiency (shallow NeRF) with high-capacity appearance modeling (deep INR).
• Address mirror symmetry in 3D-aware GANs via an auxiliary discriminator.
• Demonstrate transfer learning and 3D-aware face stylization from FFHQ to other domains.

Proposed method

• Use a shallow NeRF network to capture 3D shape with a noise-conditioned Modulated SIREN block.
• Condition NeRF with a shape code z_s mapped through a mapping network to w_s for feature modulation.
• Have NeRF output a 3D feature vector v and density sigma per point, with volume rendering to produce per-pixel features.
• Employ a deep 2D INR network to convert per-pixel features into RGB values independently for each pixel (no spatial upsampling).
• Incorporate a learnable positional encoding to avoid fixed PE-induced mirror symmetry and use an auxiliary discriminator to regularize the NeRF output.
• Implement partial gradient backpropagation to train high-resolution images efficiently by selectively enabling gradient flow over a subset of rays per iteration.
• Provide a memory-efficient ModFC implementation via batch matrix multiplication (bmm) to speed up appearance modulation.

Experimental results

Research questions

• RQ1 Can a hybrid generator (shallow NeRF + deep INR) achieve high-quality, 3D-aware image synthesis with explicit pose control?
• RQ2 Does an auxiliary discriminator effectively mitigate mirror symmetry in NeRF+INR 3D-aware GANs?
• RQ3 Can high-resolution 3D-aware GANs be trained efficiently without spatial convolutions or upsampling?
• RQ4 How well does transfer learning transfer NeRF shape knowledge while fine-tuning appearance for new domains?
• RQ5 What are the relative gains of learnable vs fixed positional encoding in mitigating symmetry issues?

Key findings

• CIPS-3D achieves state-of-the-art FID/KID among 3D-aware GANs on FFHQ, with 6.97 (2562) and 12.26 (10242) FID and 2.87 (2562) and 7.74 (10242) KID.
• Compared to StyleNeRF, CIPS-3D shows better FID/KID at 2562 despite having more parameters, while remaining competitive at higher resolutions.
• The auxiliary discriminator effectively removes mirror symmetry that arises from coordinate symmetry in NeRF-based generators.
• Partial gradient backpropagation enables training at 5122 resolution by reducing memory usage while preserving full-image discriminator exposure.
• Transfer learning is effective by freezing the NeRF (shape) network and finetuning the INR (appearance) to adapt to new domains (MetFaces, BitmojiFaces, CartoonFaces, AFHQ).
• Interpolation between base FFHQ model and transferred models yields controllable 3D-aware stylization and domain blending.