[Paper Review] StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis

Jiatao Gu, Lingjie Liu+2 more|arXiv (Cornell University)|Oct 18, 2021

Advanced Vision and Imaging45 references180 citations

TL;DR

StyleNeRF integrates a style-conditioned NeRF-based generator with progressive 2D upsampling to render high-resolution, 3D-consistent images at interactive rates, with camera control and style editing capabilities.

ABSTRACT

We propose StyleNeRF, a 3D-aware generative model for photo-realistic high-resolution image synthesis with high multi-view consistency, which can be trained on unstructured 2D images. Existing approaches either cannot synthesize high-resolution images with fine details or yield noticeable 3D-inconsistent artifacts. In addition, many of them lack control over style attributes and explicit 3D camera poses. StyleNeRF integrates the neural radiance field (NeRF) into a style-based generator to tackle the aforementioned challenges, i.e., improving rendering efficiency and 3D consistency for high-resolution image generation. We perform volume rendering only to produce a low-resolution feature map and progressively apply upsampling in 2D to address the first issue. To mitigate the inconsistencies caused by 2D upsampling, we propose multiple designs, including a better upsampler and a new regularization loss. With these designs, StyleNeRF can synthesize high-resolution images at interactive rates while preserving 3D consistency at high quality. StyleNeRF also enables control of camera poses and different levels of styles, which can generalize to unseen views. It also supports challenging tasks, including zoom-in and-out, style mixing, inversion, and semantic editing.

Motivation & Objective

• Motivate 3D-aware image synthesis that handles high-resolution outputs with strong multi-view consistency.
• Integrate a style-based generator with NeRF to improve rendering efficiency and 3D consistency.
• Enable explicit control over camera poses and style attributes, including style mixing, interpolation, inversion, and semantic editing.

Proposed method

• Model a 3D scene as a NeRF whose features are conditioned on a style vector from a mapping network.
• Predict density and color using style-conditioned MLPs, with color prediction not conditioned on view direction to improve consistency.
• Render by generating a low-resolution NeRF feature map and progressively upsampling in 2D to high resolution while preserving 3D consistency.
• Use a specialized upsampling operator that combines learnable components with fixed blur to reduce artifacts.
• Introduce NeRF-path regularization to encourage outputs to match NeRF renderings and suppress view-direction dependencies and noise injection.
• Train progressively from low to high resolution to stabilize geometry learning and multi-view consistency.

Experimental results

Research questions

• RQ1 Can a style-conditioned NeRF-based generator produce photo-realistic, high-resolution images with strong multi-view consistency?
• RQ2 Does progressive 2D upsampling with a designed upsampler preserve 3D consistency while enabling interactive rendering speeds?
• RQ3 How does removing view-direction conditioning on color and mitigating 2D noise impact 3D consistency and quality?
• RQ4 What level of explicit camera control and style manipulation (mixing, interpolation, inversion, editing) is achievable in a 3D-aware generative model?

Key findings

• StyleNeRF achieves high-quality 1024^2 image synthesis at interactive rates with strong multi-view consistency.
• The proposed upsampling design and NeRF-path regularization significantly improve 3D consistency compared to naive upsampling.
• Removing view-direction conditioning on color prediction improves synthesis consistency across views.
• The model supports explicit camera control and style manipulation, including style mixing, interpolation, inversion, and CLIP-informed editing.
• Progressive training improves stability and geometry learning over training from scratch at high resolution.