[Paper Review] Neural 3D Scene Compression via Model Compression

Berivan Isik|arXiv (Cornell University)|May 7, 2021

Advanced Vision and Imaging40 references8 citations

TL;DR

This paper proposes a novel 3D scene compression method that compresses Neural Radiance Fields (NeRF) via neural network model pruning, eliminating the need to store redundant 2D images. By pruning up to 90% of NeRF parameters and retraining, the method achieves up to 10× compression with minimal perceptual quality loss, especially at 30% pruning where PSNR improves slightly and artifacts are negligible.

ABSTRACT

Rendering 3D scenes requires access to arbitrary viewpoints from the scene. Storage of such a 3D scene can be done in two ways; (1) storing 2D images taken from the 3D scene that can reconstruct the scene back through interpolations, or (2) storing a representation of the 3D scene itself that already encodes views from all directions. So far, traditional 3D compression methods have focused on the first type of storage and compressed the original 2D images with image compression techniques. With this approach, the user first decodes the stored 2D images and then renders the 3D scene. However, this separated procedure is inefficient since a large amount of 2D images have to be stored. In this work, we take a different approach and compress a functional representation of 3D scenes. In particular, we introduce a method to compress 3D scenes by compressing the neural networks that represent the scenes as neural radiance fields. Our method provides more efficient storage of 3D scenes since it does not store 2D images -- which are redundant when we render the scene from the neural functional representation.

Motivation & Objective

• To address the inefficiency of traditional 3D compression that stores large numbers of 2D images for scene reconstruction.
• To explore functional 3D scene representations—specifically NeRFs—as a more efficient alternative to image-based storage.
• To improve 3D scene compression by applying neural network compression techniques, particularly pruning, directly to NeRF models.
• To evaluate the trade-off between compression ratio, model size, and rendering quality in NeRF-based 3D scene representation.

Proposed method

• The method compresses 3D scenes by pruning the fully connected layers of a pre-trained NeRF model, reducing model parameters without retraining initially.
• After pruning, the model is fine-tuned via retraining to recover and improve rendering quality, mitigating artifacts introduced by parameter removal.
• The approach leverages the functional nature of NeRFs, which encode a continuous 5D function (spatial coordinates and viewing direction) to output color and density, enabling direct compression of the functional representation.
• Compression ratios are measured as the ratio of original to pruned model size, with up to 10× compression achieved at 90% pruning.
• Rendering quality is evaluated using PSNR and MSE metrics on both synthetic (fern and lego) datasets, comparing original, pruned, and retrained models.
• Visual and quantitative analysis is performed on rendered views, depth maps, and mesh reconstructions to assess perceptual and structural fidelity.

Experimental results

Research questions

• RQ1 Can model pruning of NeRFs achieve significant compression without substantial degradation in 3D scene rendering quality?
• RQ2 How does retraining after pruning affect the visual and quantitative fidelity of compressed NeRF-based 3D scenes?
• RQ3 What is the trade-off between compression ratio and perceptual quality in NeRF-based 3D scene representation?
• RQ4 Does pruning induce artifacts in geometry and color, and can retraining effectively mitigate them?
• RQ5 Can functional 3D representations like NeRFs be effectively compressed using neural network compression techniques such as pruning?

Key findings

• At 30% pruning, the pruned NeRF model achieves a PSNR of 25.41 on the fern dataset and 29.14 on the lego dataset, with minimal quality degradation and even slight PSNR improvement over the original model.
• After retraining, the 30% pruned model maintains high fidelity, with PSNR values of 25.45 (fern) and 30.43 (lego), indicating effective artifact recovery.
• At 50% pruning, PSNR drops to 22.48 (fern) and 25.32 (lego), with visible artifacts in geometry and color, though retraining improves quality significantly.
• At 70% pruning, PSNR falls to 20.97 (fern) and 21.70 (lego), with noticeable degradation in mesh and depth map quality, though retraining helps restore missing parts.
• At 90% pruning, PSNR drops to 16.87 (fern) and 15.99 (lego), indicating severe quality loss, but retraining still improves reconstruction by filling in empty regions.
• The method achieves up to 10× compression ratio (at 90% pruning) with MSE values of 0.0706 (fern) and 0.0786 (lego), demonstrating high storage efficiency despite quality trade-offs.