[Paper Review] Multi-Scale 3D Gaussian Splatting for Anti-Aliased Rendering

Zhiwen Yan, Weng Fei Low+2 more|arXiv (Cornell University)|Nov 28, 2023

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes multi-scale 3D Gaussian splatting to address aliasing and rendering slowdown in low-resolution or distant-view rendering. By maintaining Gaussians at multiple scales and selectively rendering appropriate levels of detail, the method achieves 13%-66% PSNR gain and 160%-2400% speedup at 4×–128× resolution on Mip-NeRF360, with under 5% additional Gaussians and similar training time.

ABSTRACT

3D Gaussians have recently emerged as a highly efficient representation for 3D reconstruction and rendering. Despite its high rendering quality and speed at high resolutions, they both deteriorate drastically when rendered at lower resolutions or from far away camera position. During low resolution or far away rendering, the pixel size of the image can fall below the Nyquist frequency compared to the screen size of each splatted 3D Gaussian and leads to aliasing effect. The rendering is also drastically slowed down by the sequential alpha blending of more splatted Gaussians per pixel. To address these issues, we propose a multi-scale 3D Gaussian splatting algorithm, which maintains Gaussians at different scales to represent the same scene. Higher-resolution images are rendered with more small Gaussians, and lower-resolution images are rendered with fewer larger Gaussians. With similar training time, our algorithm can achieve 13\%-66\% PSNR and 160\%-2400\% rendering speed improvement at 4$ imes$-128$ imes$ scale rendering on Mip-NeRF360 dataset compared to the single scale 3D Gaussian splitting. Our code and more results are available on our project website https://jokeryan.github.io/projects/ms-gs/

Motivation & Objective

• Address severe aliasing and performance degradation in 3D Gaussian splatting when rendering at low resolutions or from distant camera positions.
• Overcome the limitation of single-scale Gaussians, which cause aliasing due to undersampling and slow rendering from excessive alpha blending at low resolution.
• Enable efficient, high-quality rendering across multiple resolution scales without increasing training time or memory usage significantly.
• Maintain high-fidelity rendering at 1× resolution while drastically improving performance and quality at lower scales.
• Introduce a scalable, selective rendering strategy inspired by mipmap and LOD techniques in computer graphics.

Proposed method

• Pre-compute and store Gaussians at multiple scales by aggregating finer Gaussians into coarser ones during training.
• Use a size-thresholding mechanism to identify small Gaussians in voxels and merge them into larger, coarser Gaussians for lower-resolution rendering.
• During rendering, dynamically select Gaussians based on pixel coverage: use finer Gaussians for high-resolution output and coarser ones for low-resolution output.
• Apply selective rendering by activating only the Gaussians whose screen-space size matches the current resolution’s pixel coverage.
• Leverage existing 3D Gaussian splatting pipeline with minimal modification, adding only a few hundred to a few thousand additional Gaussians.
• Maintain consistent training time by reusing the same optimization loop and data pipeline as single-scale 3D Gaussian splatting.

Experimental results

Research questions

• RQ1 Can multi-scale 3D Gaussians reduce aliasing artifacts in low-resolution and distant-view rendering compared to single-scale 3D Gaussian splatting?
• RQ2 Does selective rendering using scale-adaptive Gaussians improve rendering speed at lower resolutions without sacrificing image quality?
• RQ3 To what extent can performance gains be achieved with minimal additional parameters and training cost?
• RQ4 How does the method scale across different datasets and resolution levels (e.g., 4× to 128×) in terms of PSNR, LPIPS, and rendering time?
• RQ5 Can the approach maintain high-fidelity rendering at 1× resolution while significantly improving performance at lower scales?

Key findings

• On the Mip-NeRF360 dataset, the method achieves 13%-66% PSNR improvement at 4× to 128× resolution compared to single-scale 3D Gaussian splatting.
• Rendering speed increases by 160%-2400% at 128× resolution, with the largest gains observed in scenes with high geometric complexity.
• The method maintains comparable PSNR and LPIPS at 1× resolution, confirming no degradation in high-resolution quality.
• The number of additional Gaussians is less than 5% of the original set, with no significant increase in training time.
• The approach effectively mitigates aliasing by reducing over-splatting of small Gaussians in low-resolution pixels.
• On the Tank and Temple and Deep Blending datasets, the method consistently improves PSNR and reduces rendering time across all tested scales.