[Paper Review] Factor Fields: A Unified Framework for Neural Fields and Beyond

Anpei Chen, Zexiang Xu+4 more|arXiv (Cornell University)|Feb 2, 2023

Optical measurement and interference techniques16 citations

TL;DR

Factor Fields unifies neural field representations by factorizing signals into multiple coordinate-transformed factors and a projection, enabling new models like Dictionary Field (DiF). DiF improves accuracy, compactness, and training speed and supports cross-signal generalization.

ABSTRACT

We present Factor Fields, a novel framework for modeling and representing signals. Factor Fields decomposes a signal into a product of factors, each represented by a classical or neural field representation which operates on transformed input coordinates. This decomposition results in a unified framework that accommodates several recent signal representations including NeRF, Plenoxels, EG3D, Instant-NGP, and TensoRF. Additionally, our framework allows for the creation of powerful new signal representations, such as the "Dictionary Field" (DiF) which is a second contribution of this paper. Our experiments show that DiF leads to improvements in approximation quality, compactness, and training time when compared to previous fast reconstruction methods. Experimentally, our representation achieves better image approximation quality on 2D image regression tasks, higher geometric quality when reconstructing 3D signed distance fields, and higher compactness for radiance field reconstruction tasks. Furthermore, DiF enables generalization to unseen images/3D scenes by sharing bases across signals during training which greatly benefits use cases such as image regression from sparse observations and few-shot radiance field reconstruction.

Motivation & Objective

• Motivate a unified mathematical framework for neural field representations of multi-dimensional signals.
• Introduce Dictionary Field (DiF) as a two-factor decomposition (global basis + local coefficients) within Factor Fields.
• Demonstrate that sharing bases across signals improves generalization and enables few-shot reconstruction.
• Show that DiF achieves better accuracy, compactness, and training efficiency compared to prior fast reconstruction methods.

Proposed method

• Propose Factor Fields: decompose a signal into N factor fields, each f_i with its own coordinate transformation γ_i, producing a product that is mapped to the target signal via a projection P.
• Represent DiF as a two-factor instance with coefficient field c(x) and basis field b(x), using Eq. (3) to map the Hadamard product via P.
• Allow various representations for factors (polynomials, MLPs, grids, vectors) and coordinate transforms (sinusoidal, sawtooth, hashing, orthogonal) to capture multi-scale content.
• Incorporate multi-scale basis representations by concatenating transformed coordinates at multiple frequencies (PR) and use a learnable projection P (linear or shallow MLP).
• Optional indirect observations (e.g., radiance field rendering) are handled by extending P to include volumetric rendering per Eq. (6) and the standard ray integral.
• Apply space contraction to normalize coordinates and train with stochastic gradient descent, using dropout-like sparsity regularization on features.

Figure 4 : 2D Image Regression. This figure shows images represented using our DiF-Grid model. The respective image resolutions and numbers of model parameters are shown below each image. Moreover, we also report a comparison to Instant-NGP (first number) in terms of optimization time and PSNR metri

Experimental results

Research questions

• RQ1 Can Factor Fields reproduce and unify existing neural field representations (e.g., NeRF, Instant-NGP, TensoRF) under a single framework?
• RQ2 Does the Dictionary Field (DiF) provide superior accuracy, compactness, and training efficiency compared with prior methods across 2D and 3D tasks?
• RQ3 Does sharing bases across multiple signals enable generalization to unseen images or 3D scenes, particularly in few-shot or sparse-observation scenarios?
• RQ4 How do choices of factors, coordinate transformations, and projections affect performance and efficiency across reconstruction tasks?

Key findings

• DiF achieves improved approximation quality, compactness, and training time compared to previous fast reconstruction methods.
• Compared to Instant-NGP, DiF yields better reconstruction and rendering quality while halving the total model parameter count for SDF and radiance field tasks.
• DiF provides higher image regression quality, better 3D SDF geometry reconstruction, and greater radiance field compactness across evaluated tasks.
• Across-signal training with shared bases enables generalization to unseen images/3D scenes and improves reconstruction from sparse observations.
• Factor Fields can represent many existing models as special cases, and using multiple factors (N>1) enhances capacity and performance.
• DiF enables cross-signal basis learning, offering generalization benefits and efficient few-shot reconstruction.

Figure 6 : Radiance Field Reconstruction . We evaluate our DiF using NeRF-Synthetic and Tanks and Temples datasets, our method is able to reconstruct high-quality surface details.