[Paper Review] Masked Space-Time Hash Encoding for Efficient Dynamic Scene Reconstruction

Feng Wang, Zilong Chen+3 more|arXiv (Cornell University)|Oct 26, 2023

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes Masked Space-Time Hash Encoding (MSTH), a novel method for efficient dynamic 3D scene reconstruction using a learnable mask to dynamically weight a 3D hash encoding and a 4D hash encoding based on uncertainty-driven spatial-temporal dynamics. By reducing redundant 4D hash table queries in static regions, MSTH achieves state-of-the-art rendering quality with only 20 minutes of training and 130 MB memory, significantly outperforming prior methods in speed, memory efficiency, and reconstruction fidelity.

ABSTRACT

In this paper, we propose the Masked Space-Time Hash encoding (MSTH), a novel method for efficiently reconstructing dynamic 3D scenes from multi-view or monocular videos. Based on the observation that dynamic scenes often contain substantial static areas that result in redundancy in storage and computations, MSTH represents a dynamic scene as a weighted combination of a 3D hash encoding and a 4D hash encoding. The weights for the two components are represented by a learnable mask which is guided by an uncertainty-based objective to reflect the spatial and temporal importance of each 3D position. With this design, our method can reduce the hash collision rate by avoiding redundant queries and modifications on static areas, making it feasible to represent a large number of space-time voxels by hash tables with small size.Besides, without the requirements to fit the large numbers of temporally redundant features independently, our method is easier to optimize and converge rapidly with only twenty minutes of training for a 300-frame dynamic scene.As a result, MSTH obtains consistently better results than previous methods with only 20 minutes of training time and 130 MB of memory storage. Code is available at https://github.com/masked-spacetime-hashing/msth

Motivation & Objective

• To address the high computational and memory cost of 4D hash encoding in dynamic scene reconstruction due to temporal redundancy.
• To improve rendering quality and optimization efficiency by decoupling static and dynamic scene components.
• To reduce hash collisions in 4D representations by avoiding redundant updates on static regions.
• To enable rapid convergence—achieving SOTA results in just 20 minutes of training—without requiring extensive hyperparameter tuning.
• To validate robustness on realistic, complex dynamic scenes through a newly collected multi-view video dataset.

Proposed method

• MSTH represents a dynamic scene as a weighted combination of a 3D hash encoding (for static components) and a 4D hash encoding (for dynamic components), with the weights learned via a differentiable 3D mask.
• The mask is guided by an uncertainty-based objective derived from the Bayesian framework of Kendall and Gal (2017), estimating the uncertainty of each 3D point being static.
• The method maximizes the mutual information between the uncertainty estimate and the mask to ensure that low-uncertainty (static) regions are primarily encoded in the 3D hash table.
• By assigning low weights to dynamic components in static areas, MSTH reduces 4D hash table collisions and avoids redundant feature storage and updates.
• The 4D hash table size is kept comparable to a 3D table, enabling compact representation without sacrificing reconstruction quality.
• The model is trained end-to-end with a multi-view or monocular video input, using a radiance field loss to optimize view synthesis.

Experimental results

Research questions

• RQ1 Can a hybrid 3D+4D hash encoding with a learnable mask improve efficiency and quality in dynamic scene reconstruction compared to pure 4D encoding?
• RQ2 How effectively can uncertainty estimation guide the dynamic allocation of representation between static and dynamic components?
• RQ3 Can MSTH achieve SOTA performance with significantly reduced training time and memory usage compared to existing methods?
• RQ4 How does the method perform on complex, real-world dynamic scenes with large static areas and intricate motion?
• RQ5 To what extent does the uncertainty-guided mask reduce hash collisions and improve optimization stability?

Key findings

• MSTH achieves a PSNR of 29.6 on the Google Immersive dataset and 31.34 on the D-NeRF dataset, outperforming all prior methods including NeRFPlayer, HyperReel, and HexPlane.
• On the same benchmarks, MSTH reduces training time to just 20 minutes, compared to 6 hours for NeRFPlayer and 2.7 hours for HyperReel.
• The method achieves a 20% improvement in LPIPS (0.02) on the D-NeRF dataset compared to the addition-based ablation, indicating better perceptual quality.
• In qualitative comparisons, MSTH reconstructs fine motion details such as facial features, splashing sparks, and striped textures more accurately than state-of-the-art methods.
• The uncertainty-guided mask leads to a clearer, less noisy distribution, enabling better separation of static and dynamic regions and reducing hash collisions.
• The proposed method maintains high performance even on challenging scenes from the newly collected Campus dataset, with PSNR of 20.9 and SSIM of 0.722, demonstrating robustness in real-world settings.