[Paper Review] Harnessing Low-Frequency Neural Fields for Few-Shot View Synthesis

Liangchen Song, Zhong Li+5 more|arXiv (Cornell University)|Mar 15, 2023

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes HALO, a method that leverages low-frequency neural fields to regularize high-frequency neural radiance fields in few-shot view synthesis, preventing overfitting by enforcing frequency consistency in rendered images. By separating rough geometry prediction (low-frequency) from fine detail modeling (high-frequency), the approach improves generalization and reduces artifacts on both static and dynamic scenes, achieving state-of-the-art results with efficient training.

ABSTRACT

Neural Radiance Fields (NeRF) have led to breakthroughs in the novel view synthesis problem. Positional Encoding (P.E.) is a critical factor that brings the impressive performance of NeRF, where low-dimensional coordinates are mapped to high-dimensional space to better recover scene details. However, blindly increasing the frequency of P.E. leads to overfitting when the reconstruction problem is highly underconstrained, \eg, few-shot images for training. We harness low-frequency neural fields to regularize high-frequency neural fields from overfitting to better address the problem of few-shot view synthesis. We propose reconstructing with a low-frequency only field and then finishing details with a high-frequency equipped field. Unlike most existing solutions that regularize the output space (\ie, rendered images), our regularization is conducted in the input space (\ie, signal frequency). We further propose a simple-yet-effective strategy for tuning the frequency to avoid overfitting few-shot inputs: enforcing consistency among the frequency domain of rendered 2D images. Thanks to the input space regularizing scheme, our method readily applies to inputs beyond spatial locations, such as the time dimension in dynamic scenes. Comparisons with state-of-the-art on both synthetic and natural datasets validate the effectiveness of our proposed solution for few-shot view synthesis. Code is available at \href{https://github.com/lsongx/halo}{https://github.com/lsongx/halo}.

Motivation & Objective

• To address overfitting in NeRF under few-shot settings where high-frequency positional encoding leads to poor generalization.
• To exploit the complementary interpolation and extrapolation properties of low- and high-frequency neural fields for better novel view synthesis.
• To develop a data-free, input-space regularization strategy that avoids reliance on pre-trained priors or ground-truth frequency tuning.
• To extend the method to dynamic scenes by treating time as an additional input dimension, maintaining efficiency and reducing temporal flickering.
• To provide a simple yet effective frequency tuning criterion based on frequency domain consistency of rendered images.

Proposed method

• The method uses a two-stream architecture: a low-frequency-only ray-based field for predicting coarse geometry and depth, and a high-frequency point-based field for modeling fine details.
• The low-frequency ray-based field is trained to predict depth values per ray, enabling efficient and stable optimization on smooth, low-frequency signals.
• High-frequency details are added via a separate point-based radiance field, which is regularized by the low-frequency geometry prediction to prevent overfitting.
• A frequency tuning strategy is introduced that iteratively reduces positional encoding frequency until the frequency domain of rendered images remains stable under small viewpoint changes.
• The frequency consistency criterion is applied in the 2D image frequency domain, using a threshold on spectral changes to determine optimal encoding frequency.
• The framework is extendable to dynamic scenes by including time as an input dimension, with the same regularization applied to the temporal frequency domain.

Experimental results

Research questions

• RQ1 Can low-frequency neural fields effectively regularize high-frequency NeRFs to prevent overfitting in few-shot view synthesis?
• RQ2 Does input-space regularization via low-frequency signals outperform output-space regularization in terms of generalization and artifact reduction?
• RQ3 Can a frequency tuning criterion based on spectral consistency of rendered images replace human or ground-truth-guided frequency selection?
• RQ4 How does the method perform on dynamic scenes with sparse multiview and temporal observations?
• RQ5 Can the method reduce temporal flickering in dynamic NeRFs by controlling the frequency of the time dimension?

Key findings

• HALO achieves a PSNR of 25.687 on the StanfordLF dataset, significantly outperforming DietNeRF (22.815) and RegNeRF (23.995) under the 4-corner-view setting.
• On the LLFF dataset, HALO achieves a PSNR of 25.687 and an SSIM of 0.914, demonstrating strong generalization with only four training views.
• In dynamic scenes, HALO reduces temporal flickering and improves visual quality, with a 11.6% reduction in LPIPS compared to TiNeuVox-S (0.182 vs. 0.116).
• The method reduces training time to 0.67 GPU-hours per scene, significantly faster than DietNeRF (4.25) and RegNeRF (3.33).
• The frequency tuning strategy successfully identifies optimal encoding frequencies without requiring ground truth or pre-trained priors, improving robustness in low-data regimes.
• Visual comparisons show that HALO produces sharper details and fewer floating artifacts than TiNeuVox and other baselines, especially in unobserved regions and on dynamic sequences.