[Paper Review] PANeRF: Pseudo-view Augmentation for Improved Neural Radiance Fields Based on Few-shot Inputs

Young Chun Ahn, Seok-Hwan Jang+3 more|arXiv (Cornell University)|Nov 23, 2022

Advanced Vision and Imaging4 citations

TL;DR

PANeRF proposes a pseudo-view augmentation (PA) scheme that synthesizes geometrically plausible novel views from few-shot inputs to improve NeRF training. By leveraging saliency maps from DINO-ViT to separate foreground and background regions, it generates reliable pseudo-views for initial training, followed by a two-stage fine-tuning with multi-level semantic consistency (MSC) and information potential (IP) regularization, achieving state-of-the-art view synthesis quality even with 1-view inputs.

ABSTRACT

The method of neural radiance fields (NeRF) has been developed in recent years, and this technology has promising applications for synthesizing novel views of complex scenes. However, NeRF requires dense input views, typically numbering in the hundreds, for generating high-quality images. With a decrease in the number of input views, the rendering quality of NeRF for unseen viewpoints tends to degenerate drastically. To overcome this challenge, we propose pseudo-view augmentation of NeRF, a scheme that expands a sufficient amount of data by considering the geometry of few-shot inputs. We first initialized the NeRF network by leveraging the expanded pseudo-views, which efficiently minimizes uncertainty when rendering unseen views. Subsequently, we fine-tuned the network by utilizing sparse-view inputs containing precise geometry and color information. Through experiments under various settings, we verified that our model faithfully synthesizes novel-view images of superior quality and outperforms existing methods for multi-view datasets.

Motivation & Objective

• To address the degradation in NeRF rendering quality when input views are sparse, especially in real-world applications like AR/VR and autonomous driving.
• To overcome the limitations of existing few-shot NeRF methods that suffer from geometric inaccuracies, floating artifacts, and loss of detail in unseen viewpoints.
• To improve generalization and reduce uncertainty in NeRF optimization under extreme data scarcity, such as single-view input.
• To develop a robust training pipeline that combines data augmentation with semantic-aware regularization for superior view synthesis.

Proposed method

• Pseudo-view augmentation (PA) generates synthetic views by warping input images using depth maps, expanding the training data distribution around the original input views.
• Saliency maps from DINO-ViT are used to segment foreground and background regions, enabling region-wise loss computation to reduce the impact of warping artifacts.
• A two-stage training process is employed: first, the NeRF is pre-trained on all augmented pseudo-views to reduce uncertainty in novel view prediction.
• The network is then fine-tuned using ground-truth sparse views with two novel regularization losses: multi-level semantic consistency (MSC) and information potential (IP).
• MSC enforces semantic consistency at both local and global levels, preserving structural and semantic fidelity across views.
• IP minimizes uncertainty along rays by encouraging consistent radiance predictions, particularly effective under data sparsity.

Experimental results

Research questions

• RQ1 Can pseudo-view augmentation significantly improve NeRF performance when trained on only a single input view?
• RQ2 How does region-wise loss computation using saliency maps affect the reliability of pseudo-view training in the presence of warping artifacts?
• RQ3 To what extent do the proposed MSC and IP regularizers reduce uncertainty and improve geometric and semantic fidelity in novel view synthesis?
• RQ4 How does PANeRF compare to existing few-shot NeRF methods in terms of PSNR, SSIM, and LPIPS across diverse datasets and settings?

Key findings

• On the DTU dataset with 1-view input, PANeRF achieved a PSNR of 22.82, significantly outperforming baseline NeRF (20.07) and other state-of-the-art methods.
• In the narrow-baseline setting on the Realistic Synthetic 360° dataset, PANeRF achieved a PSNR of 21.50 and LPIPS of 0.156, surpassing InfoNeRF (PSNR 18.41, LPIPS 0.199).
• Ablation studies confirmed that combining PA with both MSC and IP regularization yielded the highest PSNR (22.82) and lowest LPIPS (0.144), demonstrating synergistic improvement.
• The MSC loss outperformed the SC loss from prior work, producing sharper details and better semantic consistency, as visually confirmed in the enlarged regions of Fig. 6.
• The method demonstrated robustness across multiple datasets and settings, including extreme cases like single-view and narrow-baseline inputs, maintaining high-quality synthesis.