[Paper Review] Mip-Splatting: Alias-free 3D Gaussian Splatting

Zehao Yu, Anpei Chen+3 more|arXiv (Cornell University)|Nov 27, 2023

Advanced Vision and Imaging5 citations

TL;DR

Mip-Splatting introduces a 3D smoothing filter and a 2D Mip filter to eliminate aliasing and dilation artifacts in 3D Gaussian Splatting (3DGS) when changing camera sampling rates. By constraining 3D Gaussian frequencies via the Nyquist limit and replacing 2D dilation with a Mip filter that mimics a box filter, the method enables alias-free rendering across zoom-in and zoom-out scenarios, achieving state-of-the-art performance with minimal code changes.

ABSTRACT

Recently, 3D Gaussian Splatting has demonstrated impressive novel view synthesis results, reaching high fidelity and efficiency. However, strong artifacts can be observed when changing the sampling rate, \eg, by changing focal length or camera distance. We find that the source for this phenomenon can be attributed to the lack of 3D frequency constraints and the usage of a 2D dilation filter. To address this problem, we introduce a 3D smoothing filter which constrains the size of the 3D Gaussian primitives based on the maximal sampling frequency induced by the input views, eliminating high-frequency artifacts when zooming in. Moreover, replacing 2D dilation with a 2D Mip filter, which simulates a 2D box filter, effectively mitigates aliasing and dilation issues. Our evaluation, including scenarios such a training on single-scale images and testing on multiple scales, validates the effectiveness of our approach.

Motivation & Objective

• To address severe rendering artifacts in 3D Gaussian Splatting when changing camera focal length or distance, particularly aliasing and dilation effects.
• To resolve the intrinsic shrinkage bias and frequency overflow in 3DGS that lead to high-frequency artifacts during zoom-in.
• To enable out-of-distribution generalization by training at a single scale while rendering accurately at multiple scales.
• To replace the 2D dilation filter with a Mip filter that simulates a physical imaging process’s box filter, reducing aliasing.

Proposed method

• Introduce a 3D smoothing filter that regularizes the maximum frequency of 3D Gaussian primitives based on the Nyquist-Shannon sampling theorem, derived from input view sampling rates.
• Apply a low-pass filter in 3D space during optimization to constrain high-frequency components, ensuring the 3D representation stays within the sampling limit.
• Replace the 2D dilation filter with a 2D Mip filter that approximates a 2D box filter using a Gaussian low-pass filter, effectively mitigating aliasing and dilation artifacts.
• Use a closed-form modification to the 3DGS pipeline, requiring only minor code changes and preserving compatibility with standard GPU rasterization.
• Ensure the 3D smoothing filter becomes an intrinsic part of the scene representation post-training, independent of viewpoint or sampling rate.
• Leverage the Mip filter’s multi-scale nature to enable faithful rendering at resolutions different from training, including zoom-in and zoom-out scenarios.

Experimental results

Research questions

• RQ1 Can 3D frequency constraints eliminate high-frequency artifacts in 3D Gaussian Splatting when zooming in?
• RQ2 Does replacing 2D dilation with a Mip filter reduce aliasing and dilation artifacts during zoom-out?
• RQ3 Can a single-scale training setup produce high-quality renderings at multiple sampling rates without performance degradation?
• RQ4 How does the proposed method compare to state-of-the-art methods like Mip-NeRF and Tri-MipRF in out-of-distribution generalization?
• RQ5 To what extent does the 3D smoothing filter prevent the generation of excessively small Gaussians that cause memory overflow?

Key findings

• Mip-Splatting eliminates high-frequency artifacts in zoom-in scenarios, as demonstrated by near-perfect reconstruction in 8× higher resolution renderings on the Blender dataset.
• On the Mip-NeRF 360 dataset, Mip-Splatting achieves performance on par with 3DGS and 3DGS + EWA in same-scale training, while significantly outperforming them in multi-scale testing.
• In single-scale training with multi-scale testing, Mip-Splatting maintains high fidelity at 1/4× and 1/8× downsampled resolutions, outperforming 3DGS and 3DGS + EWA.
• The method achieves alias-free rendering across scales without requiring multi-scale training data, unlike Mip-NeRF and Tri-MipRF, which rely on multi-scale input during training.
• The combination of 3D smoothing and Mip filtering prevents memory overflow from excessive small Gaussians, enabling stable training on A100 GPUs even with high-density scenes.
• Qualitative results show Mip-Splatting renderings closely resemble ground truth, especially in zoom-in scenarios, while 3DGS + EWA exhibits visible high-frequency artifacts.