[Paper Review] Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering

Tao Lu, Mulin Yu+5 more|arXiv (Cornell University)|Nov 30, 2023

Advanced Vision and Imaging4 citations

TL;DR

Scaffold-GS introduces a structured 3D Gaussian representation using anchor points from SfM to guide view-adaptive, on-the-fly prediction of Gaussian attributes (color, opacity, scale, rotation) based on viewing direction and distance. This reduces redundancy, improves robustness to view changes and texture-less regions, and achieves SOTA rendering quality at ~100 FPS with significantly reduced storage compared to 3D-GS.

ABSTRACT

Neural rendering methods have significantly advanced photo-realistic 3D scene rendering in various academic and industrial applications. The recent 3D Gaussian Splatting method has achieved the state-of-the-art rendering quality and speed combining the benefits of both primitive-based representations and volumetric representations. However, it often leads to heavily redundant Gaussians that try to fit every training view, neglecting the underlying scene geometry. Consequently, the resulting model becomes less robust to significant view changes, texture-less area and lighting effects. We introduce Scaffold-GS, which uses anchor points to distribute local 3D Gaussians, and predicts their attributes on-the-fly based on viewing direction and distance within the view frustum. Anchor growing and pruning strategies are developed based on the importance of neural Gaussians to reliably improve the scene coverage. We show that our method effectively reduces redundant Gaussians while delivering high-quality rendering. We also demonstrates an enhanced capability to accommodate scenes with varying levels-of-detail and view-dependent observations, without sacrificing the rendering speed.

Motivation & Objective

• To address the redundancy and view-inefficiency of 3D Gaussian Splatting (3D-GS), which overfits to training views and lacks robustness to large view changes.
• To improve scene coverage and rendering robustness in texture-less, reflective, or complex indoor/outdoor scenes where 3D-GS struggles.
• To reduce model storage and computational overhead while maintaining high-fidelity, real-time rendering.
• To enable efficient, on-the-fly prediction of Gaussian attributes using view-dependent MLPs conditioned on anchor features and viewing position.
• To develop a reliable anchor growing and pruning strategy that enhances scene coverage without increasing redundancy.

Proposed method

• Scaffold-GS initializes a sparse voxel grid of anchor points from SfM point clouds to serve as structural anchors for local 3D Gaussian distribution.
• Each anchor dynamically predicts Gaussian attributes (color, opacity, scale, rotation) via a view-dependent MLP using the viewing direction and distance within the view frustum.
• Neural Gaussians are filtered during inference using a learnable selector based on opacity, excluding trivial or unoccupied regions before rasterization.
• A dual-layered hierarchy is formed: sparse anchors on a grid, with a set of neural Gaussians spawned per anchor, enabling region-aware and view-adaptive representation.
• Anchor refinement is performed via growing (adding new anchors in low-coverage areas) and pruning (removing unimportant anchors), guided by predicted Gaussian importance.
• The method limits inference to anchors within the view frustum and applies opacity-based filtering to accelerate rendering with minimal fidelity loss.

Experimental results

Research questions

• RQ1 Can a structured, anchor-based 3D Gaussian representation improve view-adaptive rendering robustness in challenging scenes such as texture-less regions or high-specularity areas?
• RQ2 Can on-the-fly, view-dependent prediction of Gaussian attributes reduce redundancy and storage overhead compared to fixed 3D-GS?
• RQ3 Does a dynamic anchor refinement strategy (growing and pruning) enhance scene coverage and fidelity without increasing model size?
• RQ4 Can the method maintain high rendering speed (~100 FPS) while achieving SOTA PSNR scores with significantly reduced storage?
• RQ5 To what extent does the initial SfM point cloud quality affect performance, and can the framework compensate for sparse initialization?

Key findings

• Scaffold-GS achieves PSNR of 30.62 on the Blender Playroom dataset and 29.80 on Dr. Johnson, matching or exceeding 3D-GS performance while using 63MB and 68MB of storage respectively, a significant reduction.
• Inference speed reaches 150 FPS on the Playroom dataset with full filtering, compared to 84 FPS without filtering, demonstrating the effectiveness of opacity-based filtering.
• The method maintains high fidelity across diverse challenging conditions, including transparency, reflections, and texture-less surfaces, where 3D-GS typically fails.
• The anchor growing operation is critical for recovering fine details and texture-less regions, while pruning controls storage growth and improves efficiency.
• Experiments with varying k-values for neural Gaussian activation show convergence to a stable number of Gaussians, indicating robustness to hyperparameter settings.
• Filtering strategies have minimal impact on PSNR but significantly improve inference speed, with full filtering achieving 150 FPS on 1K resolution.