[Paper Review] SC-NeRF: Self-Correcting Neural Radiance Field with Sparse Views

Liang Song, Guangming Wang+4 more|arXiv (Cornell University)|Sep 10, 2023

Advanced Vision and Imaging4 citations

TL;DR

SC-NeRF proposes a self-correcting neural radiance field that generalizes from object-level datasets to unseen outdoor scenes using only three sparse views. By integrating multi-head attention with depth-normalized queries, it corrects geometric distortions and appearance artifacts, achieving state-of-the-art performance with PSNR of 25.989 and LPIPS of 0.224 on outdoor benchmarks.

ABSTRACT

In recent studies, the generalization of neural radiance fields for novel view synthesis task has been widely explored. However, existing methods are limited to objects and indoor scenes. In this work, we extend the generalization task to outdoor scenes, trained only on object-level datasets. This approach presents two challenges. Firstly, the significant distributional shift between training and testing scenes leads to black artifacts in rendering results. Secondly, viewpoint changes in outdoor scenes cause ghosting or missing regions in rendered images. To address these challenges, we propose a geometric correction module and an appearance correction module based on multi-head attention mechanisms. We normalize rendered depth and combine it with light direction as query in the attention mechanism. Our network effectively corrects varying scene structures and geometric features in outdoor scenes, generalizing well from object-level to unseen outdoor scenes. Additionally, we use appearance correction module to correct appearance features, preventing rendering artifacts like blank borders and ghosting due to viewpoint changes. By combining these modules, our approach successfully tackles the challenges of outdoor scene generalization, producing high-quality rendering results. When evaluated on four datasets (Blender, DTU, LLFF, Spaces), our network outperforms previous methods. Notably, compared to MVSNeRF, our network improves average PSNR from 19.369 to 25.989, SSIM from 0.838 to 0.889, and reduces LPIPS from 0.265 to 0.224 on Spaces outdoor scenes.

Motivation & Objective

• To enable generalization of neural radiance fields to outdoor scenes trained solely on object-level datasets.
• To address black artifacts and ghosting in novel view synthesis caused by distributional shift and large viewpoint changes.
• To improve rendering quality in sparse-view settings without per-scene optimization.
• To develop a differentiable, end-to-end trainable framework for robust generalization across diverse scene structures.

Proposed method

• Constructs a low-resolution 3D geometric cost volume from sparse multi-view inputs to provide continuous geometric priors.
• Introduces a geometric correction module using multi-head attention with normalized rendered depth and view direction as query.
• Employs an appearance correction module that uses the same attention mechanism to refine appearance features based on directional and depth embeddings.
• Combines both correction modules in a sequential manner: geometric correction precedes appearance correction for improved feature representation.
• Uses differentiable rendering with volume density weighting to reconstruct depth maps for evaluation.
• Leverages direction and depth embeddings as queries in multi-head attention to enhance feature alignment and reduce artifacts.

Figure 1: Comparison with previous methods IBRNet [ 12 ] and MVSNeRF [ 10 ] on Spaces. We train both their and our networks on DTU and generalize to outdoor scenes in Spaces. The left, middle, and right images respectively show the rendering result of [ 12 ] , [ 10 ] , and ours.

Experimental results

Research questions

• RQ1 Can a NeRF model trained exclusively on object-level datasets generalize effectively to complex outdoor scenes?
• RQ2 How can geometric inconsistencies caused by large viewpoint changes be mitigated in sparse-view novel view synthesis?
• RQ3 What role does depth-aware query design play in reducing rendering artifacts like ghosting and black borders?
• RQ4 To what extent does attention-based feature correction improve generalization beyond standard NeRF and IBR-based methods?

Key findings

• SC-NeRF achieves a PSNR of 25.989 on the Spaces outdoor dataset, a significant improvement over MVSNeRF’s 19.369.
• The model reduces LPIPS from 0.265 (MVSNeRF) to 0.224 on the same dataset, indicating better perceptual quality.
• SSIM improves to 0.889, demonstrating enhanced structural fidelity in rendered views.
• The ablation study confirms that combining depth and direction as attention queries reduces both white holes and shadow transfer artifacts.
• Geometric correction with depth-normalized queries reduces absolute depth error by 37% compared to MVSNeRF and 98% compared to IBRNet.
• Sequential correction—geometric first, then appearance—yields the best overall performance in PSNR, SSIM, and LPIPS metrics.

Figure 2: Overview of SC-NeRF. We first extract image features and warp them onto a plane sweep,then use 3DCNN to build a geometric Volume. Second, we use the geometric feature rectification and appearance feature rectification modules to obtain the final radiation features. Finally, we use an MLP t