[Paper Review] Tri-MipRF: Tri-Mip Representation for Efficient Anti-Aliasing Neural Radiance Fields

Wenbo Hu, Yuling Wang+5 more|arXiv (Cornell University)|Jul 21, 2023

Advanced Neuroimaging Techniques and Applications4 citations

TL;DR

Tri-MipRF proposes a novel Tri-Mip encoding that factorizes 3D pre-filtered feature spaces into three orthogonal 2D mipmaps, enabling efficient area sampling for anti-aliased neural radiance fields. By combining this encoding with cone-casting rendering and a hybrid volume-surface strategy, it achieves state-of-the-art rendering quality with sub-5-minute reconstruction and 25% smaller model size than Instant-NGP.

ABSTRACT

Despite the tremendous progress in neural radiance fields (NeRF), we still face a dilemma of the trade-off between quality and efficiency, e.g., MipNeRF presents fine-detailed and anti-aliased renderings but takes days for training, while Instant-ngp can accomplish the reconstruction in a few minutes but suffers from blurring or aliasing when rendering at various distances or resolutions due to ignoring the sampling area. To this end, we propose a novel Tri-Mip encoding that enables both instant reconstruction and anti-aliased high-fidelity rendering for neural radiance fields. The key is to factorize the pre-filtered 3D feature spaces in three orthogonal mipmaps. In this way, we can efficiently perform 3D area sampling by taking advantage of 2D pre-filtered feature maps, which significantly elevates the rendering quality without sacrificing efficiency. To cope with the novel Tri-Mip representation, we propose a cone-casting rendering technique to efficiently sample anti-aliased 3D features with the Tri-Mip encoding considering both pixel imaging and observing distance. Extensive experiments on both synthetic and real-world datasets demonstrate our method achieves state-of-the-art rendering quality and reconstruction speed while maintaining a compact representation that reduces 25% model size compared against Instant-ngp.

Motivation & Objective

• Address the long-standing trade-off between rendering quality and reconstruction efficiency in neural radiance fields.
• Overcome aliasing in distant views and blurring in close-up views caused by point-based sampling in efficient methods like Instant-NGP.
• Enable high-fidelity, anti-aliased rendering without sacrificing reconstruction speed or model compactness.
• Design a representation that supports efficient area sampling for 3D features while remaining memory-efficient and compatible with fast training.

Proposed method

• Propose a Tri-Mip encoding that decomposes 3D space into three orthogonal planes (XY, XZ, YZ), each represented as a mipmap to model pre-filtered 3D features.
• Use 2D pre-filtered mipmaps to enable efficient 3D area sampling by leveraging isotropic pre-filtering across mipmap levels.
• Introduce cone-casting rendering where each pixel is treated as a disc, and cones are cast with inscribed spheres for adaptive sampling.
• Featurize sampling spheres using the Tri-Mip encoding based on their occupied area in the 3D space, ensuring anti-aliasing across distances.
• Develop a hybrid volume-surface rendering strategy to achieve real-time inference (>60 FPS) on consumer GPUs.
• Combine the shallow MLP with the compact Tri-Mip encoding to maintain a small model size (48.2 MB) while enabling fast convergence.

Experimental results

Research questions

• RQ1 Can a compact 3D representation be designed to support both efficient reconstruction and anti-aliased rendering?
• RQ2 How can pre-filtered 3D features be efficiently modeled using 2D mipmap structures without incurring high memory or computational costs?
• RQ3 Can cone-casting with area-based sampling using pre-filtered 2D maps eliminate aliasing and blurring across varying viewing distances?
• RQ4 Is it possible to achieve state-of-the-art rendering quality with reconstruction times under five minutes and model sizes smaller than existing hybrid methods?
• RQ5 How does the proposed method perform on real-world in-the-wild captures with complex geometry and appearance?

Key findings

• Tri-MipRF achieves state-of-the-art PSNR (33.65) and SSIM (0.963) on the single-scale Blender dataset, outperforming MipNeRF, TensoRF, and Instant-NGP.
• The method reduces model size by 25% compared to Instant-NGP, achieving 48.2 MB storage, which is the smallest among hybrid methods.
• Reconstruction is completed in under five minutes on a single GPU, significantly faster than MipNeRF (3+ days) and faster than the 5× super-sampling variant of Instant-NGP.
• On in-the-wild captures, Tri-MipRF produces high-fidelity renderings with fine details, even surpassing ground truth in some cases due to artifact suppression during multi-view fusion.
• The method enables real-time rendering at over 60 FPS on an NVIDIA RTX 3060, demonstrating practical usability on consumer hardware.
• Visual results show no aliasing in distant views and no blurring in close-ups, confirming the effectiveness of cone-casting with area-based sampling.