[Paper Review] Mixed Neural Voxels for Fast Multi-view Video Synthesis

Feng Wang, Sinan Tan+4 more|arXiv (Cornell University)|Dec 1, 2022

Advanced Vision and Imaging4 citations

TL;DR

MixVoxels proposes a mixed static-dynamic voxel representation with separate networks for static and dynamic regions, enabling 15-minute training for 300-frame videos and 37 fps rendering at 1K resolution. It uses an inner product time query for efficient multi-time-step supervision and a variation field to distinguish static from dynamic voxels, achieving SOTA PSNR with 5000× faster training than implicit methods.

ABSTRACT

Synthesizing high-fidelity videos from real-world multi-view input is challenging because of the complexities of real-world environments and highly dynamic motions. Previous works based on neural radiance fields have demonstrated high-quality reconstructions of dynamic scenes. However, training such models on real-world scenes is time-consuming, usually taking days or weeks. In this paper, we present a novel method named MixVoxels to better represent the dynamic scenes with fast training speed and competitive rendering qualities. The proposed MixVoxels represents the 4D dynamic scenes as a mixture of static and dynamic voxels and processes them with different networks. In this way, the computation of the required modalities for static voxels can be processed by a lightweight model, which essentially reduces the amount of computation, especially for many daily dynamic scenes dominated by the static background. To separate the two kinds of voxels, we propose a novel variation field to estimate the temporal variance of each voxel. For the dynamic voxels, we design an inner-product time query method to efficiently query multiple time steps, which is essential to recover the high-dynamic motions. As a result, with 15 minutes of training for dynamic scenes with inputs of 300-frame videos, MixVoxels achieves better PSNR than previous methods. Codes and trained models are available at https://github.com/fengres/mixvoxels

Motivation & Objective

• To address the high computational cost and slow training of neural radiance fields for real-world dynamic scenes.
• To improve rendering quality for high-dynamic movements while reducing redundant computation on static backgrounds.
• To enable fast, practical training and inference for multi-view video synthesis in complex real-world environments.
• To eliminate reliance on complex importance sampling by separating static and dynamic components.

Proposed method

• MixVoxels represents 4D scenes as a mixture of static and dynamic voxels, processed by separate neural networks to reduce computation on static regions.
• A variation field estimates temporal variance per voxel to classify it as static or dynamic, enabling spatial separation of components.
• An inner product time query mechanism efficiently queries multiple time steps simultaneously, improving temporal modeling for high-dynamic objects.
• The method uses coordinate-based neural networks with shared weights across time, enabling fast inference and training.
• Training is accelerated by applying lightweight models to static voxels and focusing dynamic models on motion-prone regions.
• The framework avoids complex ray sampling strategies by leveraging the static-dynamic split to improve motion sharpness.

Figure 1: Our method enables rapid reconstruction of 4D dynamic scenes. We visualize the rendering results with different training schedules. With only 15 minutes of training, our approach achieves comparable PSNRs to other methods. Increasing the training time further enhances the ability to recove

Experimental results

Research questions

• RQ1 Can a mixed static-dynamic voxel representation significantly reduce training time without sacrificing rendering quality?
• RQ2 How does separating static and dynamic components affect the model's ability to capture sharp motion details?
• RQ3 Can an efficient time query mechanism like inner product querying improve performance on high-dynamic scenes compared to concatenation or Fourier-based methods?
• RQ4 To what extent does the variation field enable accurate static-dynamic voxel classification in real-world scenes?

Key findings

• MixVoxels achieves a PSNR of 29.47 after 0.6 hours of training, outperforming the full-dynamic baseline (28.36 PSNR) despite significantly faster training.
• The method renders novel view videos at 37 fps with 1K resolution, enabling real-time interactive applications.
• With only 15 minutes of training, MixVoxels achieves PSNR comparable to prior methods trained for much longer durations.
• The inner product time query method reduces training time to 40 minutes (vs. 58 minutes for concatenation) and improves PSNR by 0.52 over concatenation.
• Ablation shows that the mixed voxel approach improves PSNR by 1.11 and reduces DSSIM by 0.010, confirming the benefit of component separation.
• The method reduces training time by 5000× compared to implicit dynamic scene representations while maintaining or exceeding SOTA performance.

Figure 2: Overview of our method. Given a ray, we first sample points, and split them into static and dynamic ones using the variation field. After that, we feed these points to the corresponding branches and query the required properties. Then we merge the static output and dynamic output for rende