[Paper Review] ResFields: Residual Neural Fields for Spatiotemporal Signals

Marko Mihajlović, Sergey Prokudin+2 more|arXiv (Cornell University)|Sep 6, 2023

3D Shape Modeling and Analysis4 citations

TL;DR

ResFields introduces a novel architecture that enhances multi-layer perceptrons (MLPs) for spatiotemporal signal modeling by integrating time-dependent residual weights into standard layers, enabling increased model capacity without expanding network width. This approach improves performance in dynamic 3D reconstruction, video approximation, and temporal signed distance fields while maintaining fast inference and reduced memory usage, achieving state-of-the-art results on challenging benchmarks with minimal parameter growth.

ABSTRACT

Neural fields, a category of neural networks trained to represent high-frequency signals, have gained significant attention in recent years due to their impressive performance in modeling complex 3D data, such as signed distance (SDFs) or radiance fields (NeRFs), via a single multi-layer perceptron (MLP). However, despite the power and simplicity of representing signals with an MLP, these methods still face challenges when modeling large and complex temporal signals due to the limited capacity of MLPs. In this paper, we propose an effective approach to address this limitation by incorporating temporal residual layers into neural fields, dubbed ResFields. It is a novel class of networks specifically designed to effectively represent complex temporal signals. We conduct a comprehensive analysis of the properties of ResFields and propose a matrix factorization technique to reduce the number of trainable parameters and enhance generalization capabilities. Importantly, our formulation seamlessly integrates with existing MLP-based neural fields and consistently improves results across various challenging tasks: 2D video approximation, dynamic shape modeling via temporal SDFs, and dynamic NeRF reconstruction. Lastly, we demonstrate the practical utility of ResFields by showcasing its effectiveness in capturing dynamic 3D scenes from sparse RGBD cameras of a lightweight capture system.

Motivation & Objective

• To address the limited capacity of standard MLP-based neural fields in modeling complex, high-frequency spatiotemporal signals such as long videos or dynamic 3D scenes.
• To enhance model capacity without increasing network width or depth, preserving fast inference and low memory usage.
• To maintain implicit regularization and generalization properties inherent in standard MLPs, unlike partitioned or meta-learned approaches.
• To enable seamless integration with existing MLP-based neural field frameworks, including NeRF and SDF-based methods.
• To reduce overfitting and improve generalization through a novel matrix factorization technique for residual weights.

Proposed method

• Replace standard linear layers in an MLP with time-dependent residual layers, where the weights are modeled as a learnable residual component added to the original weights: $\mathbf{W}_i(t) = \mathbf{W}_i + \bm{\mathcal{W}}_i(t)$.
• Introduce a matrix factorization technique to parameterize the time-dependent residual weights $\bm{\mathcal{W}}_i(t)$ as a low-rank decomposition, reducing the number of trainable parameters.
• Use a factorization scheme $\bm{\mathcal{W}}_i(t) = \mathbf{v}_i(t) \mathbf{M}_i$, where $\mathbf{v}_i(t) \in \mathbb{R}^{N_i \times R_i}$ and $\mathbf{M}_i \in \mathbb{R}^{R_i \times M_i}$, to achieve parameter efficiency.
• Maintain compatibility with existing neural field frameworks such as SIREN and TNeRF by directly plugging the ResField layers into their architectures.
• Apply positional encoding and spectral bias mitigation techniques (e.g., SIREN) in conjunction with ResFields to further improve modeling of high-frequency signals.
• Train the full network end-to-end using standard optimization, with the residual components learning to modulate the base network’s capacity over time.

Figure 2: ResField MLP Architecture.

Experimental results

Research questions

• RQ1 Can residual learning be effectively extended to the weight space of MLPs in neural fields to increase modeling capacity for temporal signals?
• RQ2 Does the proposed ResField architecture maintain the implicit regularization and generalization properties of standard MLPs while improving performance on complex spatiotemporal tasks?
• RQ3 Can matrix factorization of residual weights significantly reduce parameter count without degrading performance, especially in low-data regimes?
• RQ4 How does ResFields compare to existing methods like partitioned neural fields or meta-learned weights in terms of reconstruction quality, training speed, and memory efficiency?
• RQ5 To what extent can ResFields improve performance on real-world dynamic 3D capture from sparse RGBD cameras?

Key findings

• ResFields achieve state-of-the-art performance on dynamic NeRF reconstruction from three time-synchronized RGBD cameras, outperforming baseline methods in both PSNR and LPIPS metrics.
• On the video approximation task, ResFields with factorization achieve a PSNR of 42.45 on 30% of unseen pixels, outperforming the baseline and other residual variants.
• The proposed matrix factorization reduces the number of trainable parameters by up to 60% compared to non-factorized ResFields while maintaining or improving generalization.
• ResFields maintain fast inference and training speed, with GPU memory usage remaining low even when modeling long and complex temporal signals.
• The method achieves consistent improvements across diverse tasks: 2D video approximation, temporal SDFs, and dynamic NeRF, demonstrating broad applicability.
• The ablation study confirms that residual weight modulation is more effective than residual output or multiplicative modulation, and that factorization significantly improves generalization over unstructured residual parameters.

Figure 3: Factorization of $\bm{\mathcal{W}}_{i}$ .