[Paper Review] SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image

Dejia Xu, Yifan Jiang+4 more|arXiv (Cornell University)|Apr 2, 2022

Image Enhancement Techniques4 citations

TL;DR

SinNeRF proposes a semi-supervised framework to train Neural Radiance Fields from a single image in complex scenes, using geometry and semantic pseudo labels to enforce multi-view consistency and perceptual quality. It achieves state-of-the-art novel-view synthesis results without multi-view supervision or pre-training on large datasets.

ABSTRACT

Despite the rapid development of Neural Radiance Field (NeRF), the necessity of dense covers largely prohibits its wider applications. While several recent works have attempted to address this issue, they either operate with sparse views (yet still, a few of them) or on simple objects/scenes. In this work, we consider a more ambitious task: training neural radiance field, over realistically complex visual scenes, by "looking only once", i.e., using only a single view. To attain this goal, we present a Single View NeRF (SinNeRF) framework consisting of thoughtfully designed semantic and geometry regularizations. Specifically, SinNeRF constructs a semi-supervised learning process, where we introduce and propagate geometry pseudo labels and semantic pseudo labels to guide the progressive training process. Extensive experiments are conducted on complex scene benchmarks, including NeRF synthetic dataset, Local Light Field Fusion dataset, and DTU dataset. We show that even without pre-training on multi-view datasets, SinNeRF can yield photo-realistic novel-view synthesis results. Under the single image setting, SinNeRF significantly outperforms the current state-of-the-art NeRF baselines in all cases. Project page: https://vita-group.github.io/SinNeRF/

Motivation & Objective

• To enable training of Neural Radiance Fields on complex scenes using only a single reference image, overcoming the need for dense multi-view coverage.
• To address the challenge of 3D geometry reconstruction and view synthesis from a single image, where prior methods fail due to lack of depth and structural priors.
• To develop a semi-supervised training strategy that leverages pseudo labels to stabilize and guide the training process in the absence of multi-view supervision.
• To achieve photo-realistic novel-view synthesis without pre-training on large-scale multi-view datasets or reliance on specialized architectures.

Proposed method

• Introduces a semi-supervised training framework that generates and propagates geometry pseudo labels via image warping between the reference view and unseen views to ensure multi-view geometry consistency.
• Employs a discriminator and a pre-trained Vision Transformer (ViT) to generate semantic pseudo labels that guide local texture quality and global structural fidelity in novel views.
• Uses a multi-loss training objective combining geometry supervision, adversarial loss for local texture realism, and a [CLS] token-based contrastive loss from DINO-ViT for global semantic consistency.
• Applies depth supervision only on the reference view, while using pseudo-labeled geometry and semantic priors to regularize unseen views during training.
• Leverages image warping to reproject depth from the reference view to unseen views, enabling consistent 3D geometry across views without explicit multi-view supervision.
• Utilizes a progressive training strategy where pseudo labels are iteratively refined and propagated to improve generalization and reduce mode collapse.

Experimental results

Research questions

• RQ1 Can a neural radiance field be effectively trained from a single image in complex scenes without multi-view supervision?
• RQ2 How can geometry and semantic priors be synthesized from a single image to stabilize NeRF training?
• RQ3 What is the relative contribution of geometry vs. semantic pseudo labels in improving novel-view synthesis quality?
• RQ4 Can a semi-supervised framework with pseudo labels outperform existing SOTA methods in single-image NeRF training?

Key findings

• SinNeRF achieves state-of-the-art performance on complex scene benchmarks, including NeRF synthetic, LLFF, and DTU datasets, without pre-training on multi-view data.
• On the DTU dataset, SinNeRF achieves a PSNR of 20.97, SSIM of 0.82, and LPIPS of 0.0932, outperforming DS-NeRF, PixelNeRF, and DietNeRF in both quantitative and qualitative results.
• Ablation studies show that removing the geometry pseudo label leads to significant geometric errors, while removing the semantic pseudo label causes structural artifacts and incorrect global layout.
• The use of DINO-ViT’s [CLS] token for global structure prior yields superior performance compared to VGG-based content, style, or self-similarity losses, demonstrating robustness to pixel misalignment.
• The variant without adversarial training suffers from blurry artifacts, confirming the importance of local texture guidance via the discriminator.
• The full model with all components achieves the best results, with a 2.77 PSNR gain over the variant without the global structure prior and a 0.0987 LPIPS improvement over the baseline.