[Paper Review] Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-time

Liao Wang, Jiakai Zhang+7 more|arXiv (Cornell University)|Feb 17, 2022

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes Fourier PlenOctrees (FPO), a novel neural representation that enables real-time, high-fidelity rendering of dynamic scenes from multi-view video sequences. By combining generalized NeRF, PlenOctree data structures, volumetric fusion, and Fourier encoding of time-varying color and density, FPO achieves 3000× speedup over original NeRF and over 10× faster inference than SOTA methods while supporting efficient fine-tuning and compact memory usage.

ABSTRACT

Implicit neural representations such as Neural Radiance Field (NeRF) have focused mainly on modeling static objects captured under multi-view settings where real-time rendering can be achieved with smart data structures, e.g., PlenOctree. In this paper, we present a novel Fourier PlenOctree (FPO) technique to tackle efficient neural modeling and real-time rendering of dynamic scenes captured under the free-view video (FVV) setting. The key idea in our FPO is a novel combination of generalized NeRF, PlenOctree representation, volumetric fusion and Fourier transform. To accelerate FPO construction, we present a novel coarse-to-fine fusion scheme that leverages the generalizable NeRF technique to generate the tree via spatial blending. To tackle dynamic scenes, we tailor the implicit network to model the Fourier coefficients of timevarying density and color attributes. Finally, we construct the FPO and train the Fourier coefficients directly on the leaves of a union PlenOctree structure of the dynamic sequence. We show that the resulting FPO enables compact memory overload to handle dynamic objects and supports efficient fine-tuning. Extensive experiments show that the proposed method is 3000 times faster than the original NeRF and achieves over an order of magnitude acceleration over SOTA while preserving high visual quality for the free-viewpoint rendering of unseen dynamic scenes.

Motivation & Objective

• To enable real-time, photorealistic free-viewpoint rendering of dynamic scenes from multi-view video inputs.
• To address the limitations of existing neural rendering methods that are too slow or not generalizable for non-rigid, dynamic objects.
• To reduce memory overhead and training time while preserving high visual quality in dynamic scene modeling.
• To support efficient fine-tuning for new dynamic scenes without retraining from scratch.
• To develop a scalable, compact neural representation suitable for VR, AR, and immersive telepresence applications.

Proposed method

• A coarse-to-fine fusion scheme leverages generalizable NeRF to rapidly generate a PlenOctree from multi-view images using sparse view rendering and spatial blending.
• The method constructs a union PlenOctree over the entire dynamic sequence to represent spatiotemporal geometry and appearance.
• Time-varying density and color are modeled as Fourier coefficients in the temporal domain, enabling compact, frequency-based representation.
• The implicit network is trained directly on the leaves of the union PlenOctree to predict Fourier coefficients of plenoptic functions.
• High-frequency Fourier bases are discarded to reduce storage while preserving perceptual detail.
• A novel volumetric fusion strategy with spatial blending enables fast, accurate PlenOctree construction in minutes.

Figure 1 : Our method achieves a photo-realistic rendered result for dynamic scenes in real-time based on our novel Fourier PlenOctree structure.

Experimental results

Research questions

• RQ1 Can a neural representation be designed to enable real-time rendering of dynamic scenes while maintaining high visual fidelity?
• RQ2 How can Fourier encoding of time-varying attributes improve memory efficiency and rendering speed in neural radiance fields?
• RQ3 Can a coarse-to-fine fusion pipeline based on generalizable NeRF accelerate PlenOctree construction for dynamic scenes?
• RQ4 What is the optimal trade-off between Fourier dimensionality, storage, and rendering quality in dynamic scene modeling?
• RQ5 To what extent does fine-tuning improve rendering quality, and how quickly can convergence be achieved?

Key findings

• FPO achieves 3000× speedup over the original NeRF implementation, enabling real-time rendering of dynamic scenes.
• The method achieves 117.87 FPS with PSNR of 36.21 using a Fourier dimension of n₁=31, n₂=5, balancing quality, speed, and memory.
• 10 minutes of fine-tuning yields near-optimal visual quality, with no significant improvement after 10 hours of training.
• Using DFT with limited memory (7.25 GB) outperforms non-DFT methods under memory constraints, achieving PSNR of 36.21 vs. 32.15 and 25.97.
• The FPO model uses only 7.25 GB of storage and enables real-time rendering with minimal fine-tuning time, outperforming both memory-limited and unlimited baselines.
• The approach supports efficient fine-tuning and compact memory overload, making it suitable for dynamic scene rendering in VR/AR and telepresence.

Figure 2 : Illustration of our fast PlenOctree generation for static scene and fast Fourier PlenOctree generation for dynamic scene. (a) illustrates how to generate a PlenOctree from multiview images. Given these images, a generalized NeRF $\Psi$ predicts view-dependent density $\sigma$ and color $\