[Paper Review] Octree-GS: Towards Consistent Real-time Rendering with LOD-Structured 3D Gaussians

Kerui Ren, Lihan Jiang+5 more|arXiv (Cornell University)|Mar 26, 2024

Computer Graphics and Visualization Techniques4 citations

TL;DR

Octree-GS introduces an LOD-structured 3D Gaussian splatting framework using hierarchical octree-based anchor points to enable consistent real-time rendering across varying view distances. By dynamically selecting multi-resolution Gaussians based on viewing distance and scene complexity, it achieves stable performance above 30 FPS even in large, detail-rich scenes like MatrixCity, outperforming SOTA methods such as Mip-Splatting and Scaffold-GS in both visual quality and rendering consistency.

ABSTRACT

The recent 3D Gaussian splatting (3D-GS) has shown remarkable rendering fidelity and efficiency compared to NeRF-based neural scene representations. While demonstrating the potential for real-time rendering, 3D-GS encounters rendering bottlenecks in large scenes with complex details due to an excessive number of Gaussian primitives located within the viewing frustum. This limitation is particularly noticeable in zoom-out views and can lead to inconsistent rendering speeds in scenes with varying details. Moreover, it often struggles to capture the corresponding level of details at different scales with its heuristic density control operation. Inspired by the Level-of-Detail (LOD) techniques, we introduce Octree-GS, featuring an LOD-structured 3D Gaussian approach supporting level-of-detail decomposition for scene representation that contributes to the final rendering results. Our model dynamically selects the appropriate level from the set of multi-resolution anchor points, ensuring consistent rendering performance with adaptive LOD adjustments while maintaining high-fidelity rendering results.

Motivation & Objective

• Address the inconsistency in rendering speed and visual quality of 3D Gaussian Splatting (3D-GS) in large-scale, detail-rich scenes during zoom-out trajectories.
• Overcome the limitations of view-frustum-only Gaussian selection, which causes redundant computation and poor LOD adaptation.
• Introduce a hierarchical, multi-resolution octree structure to organize 3D Gaussians for level-of-detail (LOD) aware rendering.
• Enable dynamic LOD selection during inference to balance performance and fidelity without increasing the number of Gaussians.
• Improve rendering quality in distant and high-frequency detail regions through progressive training and LOD bias mechanisms.

Proposed method

• Structure 3D Gaussians using a hierarchical octree data structure, where each node represents a multi-resolution anchor point for Gaussians.
• Dynamically select the appropriate LOD level based on viewing distance and scene detail richness, reducing redundant rendering of distant or low-visibility features.
• Introduce a next-level grow operator to propagate Gaussians from coarser to finer levels, enabling cross-level interaction and improved detail capture.
• Apply an LOD bias mechanism to prioritize high-frequency details in distant views, reducing visual artifacts and edge blurring.
• Employ progressive training starting from coarser LOD levels (e.g., level 2 or 4) to stabilize optimization and prevent overlap between levels.
• Integrate the octree-structured Gaussians into the Scaffold-GS framework, preserving its view-adaptive and differentiable rendering pipeline.

Experimental results

Research questions

• RQ1 Can a hierarchical, LOD-aware 3D Gaussian representation improve rendering consistency across varying view distances in large scenes?
• RQ2 How does dynamic LOD selection based on viewing distance affect rendering performance and visual quality compared to fixed-view-frustum selection?
• RQ3 To what extent does progressive training from coarse to fine levels enhance the alignment of Gaussians with scene geometry and reduce artifacts?
• RQ4 Can an octree-based anchor structure effectively manage multi-resolution Gaussians while maintaining real-time rendering speed?
• RQ5 How does the proposed LOD bias mechanism improve edge sharpness and detail preservation in distant views?

Key findings

• Octree-GS achieves consistent real-time rendering at or above 30 FPS across all zoom-out trajectories in the MatrixCity dataset, unlike Mip-Splatting and Scaffold-GS, which drop below 30 FPS at distance.
• The model maintains high visual fidelity, with PSNR of 31.05 and LPIPS of 0.055 on the MatrixCity dataset, outperforming SOTA baselines in both metrics.
• Ablation studies show that adding the next-level grow operator increases PSNR by 0.47 (from 28.78 to 29.25) and reduces LPIPS by 0.025, demonstrating improved detail capture.
• Incorporating LOD bias improves PSNR by 0.21 and reduces LPIPS by 0.002 compared to the baseline, significantly enhancing edge sharpness in distant views.
• Progressive training starting from level 2 or 4 improves rendering accuracy of coarse levels, with M7 (init @4) achieving 29.50 PSNR and 0.177 LPIPS, showing better fidelity than non-progressive variants.
• The full model (with all components) achieves 31.05 PSNR and 0.055 LPIPS, demonstrating that the combination of octree structure, LOD bias, and progressive training yields optimal performance.