[论文解读] Factor Fields: A Unified Framework for Neural Fields and Beyond

Anpei Chen, Zexiang Xu+4 more|arXiv (Cornell University)|Feb 2, 2023

Optical measurement and interference techniques被引用 16

一句话总结

Factor Fields 通过将信号分解为多个坐标变换因子和一个投影来统一神经场表示，催生了如 Dictionary Field (DiF) 这类新模型。DiF 提升了准确性、紧凑性和训练速度，并支持跨信号的泛化。

ABSTRACT

We present Factor Fields, a novel framework for modeling and representing signals. Factor Fields decomposes a signal into a product of factors, each represented by a classical or neural field representation which operates on transformed input coordinates. This decomposition results in a unified framework that accommodates several recent signal representations including NeRF, Plenoxels, EG3D, Instant-NGP, and TensoRF. Additionally, our framework allows for the creation of powerful new signal representations, such as the "Dictionary Field" (DiF) which is a second contribution of this paper. Our experiments show that DiF leads to improvements in approximation quality, compactness, and training time when compared to previous fast reconstruction methods. Experimentally, our representation achieves better image approximation quality on 2D image regression tasks, higher geometric quality when reconstructing 3D signed distance fields, and higher compactness for radiance field reconstruction tasks. Furthermore, DiF enables generalization to unseen images/3D scenes by sharing bases across signals during training which greatly benefits use cases such as image regression from sparse observations and few-shot radiance field reconstruction.

研究动机与目标

• 为多维信号的神经场表示提供一个统一的数学框架。
• 在 Factor Fields 内将 Dictionary Field (DiF) 引入为一个两因素分解（全局基底 + 局部系数）。
• 证明跨信号共享基底可以提升泛化能力并实现少样本重建。
• 显示 DiF 相比先前的快速重建方法在准确性、紧凑性和训练效率方面具备优势。

提出的方法

• 提出 Factor Fields：将信号分解为 N 个因子场，每个 f_i 拥有自身的坐标变换 γ_i，生成的积通过投影 P 映射到目标信号。
• 将 DiF 表示为一个具有系数场 c(x) 和基底场 b(x) 的两因素实例，使用 Eq. (3) 将 Hadamard 积通过 P 映射。
• 允许因子具有多种表示形式（多项式、MLP、网格、向量）和坐标变换（正弦、锯齿、哈希、正交），以捕捉多尺度内容。
• 通过在多个频率处连接变换后的坐标来包含多尺度基表示（PR），并使用一个可学习的投影 P（线性或浅层 MLP）。
• 可选的间接观测（如辐射场渲染）通过将 P 扩展以包含按 Eq. (6) 和标准射线积分的体积渲染来处理。
• 应用空间收缩以归一化坐标，并使用随机梯度下降训练，对特征使用类似 dropout 的稀疏正则化。

Figure 4 : 2D Image Regression. This figure shows images represented using our DiF-Grid model. The respective image resolutions and numbers of model parameters are shown below each image. Moreover, we also report a comparison to Instant-NGP (first number) in terms of optimization time and PSNR metri

实验结果

研究问题

• RQ1 Can Factor Fields reproduce and unify existing neural field representations (e.g., NeRF、Instant-NGP、TensoRF) under a single framework?
• RQ2 Does the Dictionary Field (DiF) provide superior accuracy, compactness, and training efficiency compared with prior methods across 2D and 3D tasks?
• RQ3 Does sharing bases across multiple signals enable generalization to unseen images or 3D scenes, particularly in few-shot or sparse-observation scenarios?
• RQ4 How do choices of factors, coordinate transformations, and projections affect performance and efficiency across reconstruction tasks?

主要发现

• DiF achieves improved approximation quality, compactness, and training time compared to previous fast reconstruction methods.
• Compared to Instant-NGP, DiF yields better reconstruction and rendering quality while halving the total model parameter count for SDF and radiance field tasks.
• DiF provides higher image regression quality, better 3D SDF geometry reconstruction, and greater radiance field compactness across evaluated tasks.
• Across-signal training with shared bases enables generalization to unseen images/3D scenes and improves reconstruction from sparse observations.
• Factor Fields can represent many existing models as special cases, and using multiple factors (N>1) enhances capacity and performance.
• DiF enables cross-signal basis learning, offering generalization benefits and efficient few-shot reconstruction.

Figure 6 : Radiance Field Reconstruction . We evaluate our DiF using NeRF-Synthetic and Tanks and Temples datasets, our method is able to reconstruct high-quality surface details.