[Paper Review] R$^2$-Gaussian: Rectifying Radiative Gaussian Splatting for Tomographic Reconstruction

Ruyi Zha, Lin Tao+4 more|arXiv (Cornell University)|May 31, 2024

Medical Imaging Techniques and ApplicationsMedicine3 citations

TL;DR

R²-Gaussian proposes the first 3D Gaussian Splatting-based method for sparse-view tomographic reconstruction, rectifying an overlooked integration bias in standard 3DGS via tailored X-ray rasterization and a differentiable CUDA voxelizer. It achieves state-of-the-art performance with 38.90 dB PSNR and 12× faster inference than NeRF-based methods, converging in under 9 minutes.

ABSTRACT

3D Gaussian splatting (3DGS) has shown promising results in image rendering and surface reconstruction. However, its potential in volumetric reconstruction tasks, such as X-ray computed tomography, remains under-explored. This paper introduces R$^2$-Gaussian, the first 3DGS-based framework for sparse-view tomographic reconstruction. By carefully deriving X-ray rasterization functions, we discover a previously unknown integration bias in the standard 3DGS formulation, which hampers accurate volume retrieval. To address this issue, we propose a novel rectification technique via refactoring the projection from 3D to 2D Gaussians. Our new method presents three key innovations: (1) introducing tailored Gaussian kernels, (2) extending rasterization to X-ray imaging, and (3) developing a CUDA-based differentiable voxelizer. Experiments on synthetic and real-world datasets demonstrate that our method outperforms state-of-the-art approaches in accuracy and efficiency. Crucially, it delivers high-quality results in 4 minutes, which is 12$ imes$ faster than NeRF-based methods and on par with traditional algorithms. Code and models are available on the project page https://github.com/Ruyi-Zha/r2_gaussian.

Motivation & Objective

• To address the lack of effective 3D Gaussian Splatting (3DGS) methods for volumetric tomographic reconstruction, particularly in sparse-view X-ray CT.
• To identify and resolve a previously unknown integration bias in standard 3DGS that degrades accuracy in density field retrieval for tomography.
• To develop a differentiable, X-ray-specific rasterization pipeline that enables end-to-end optimization of 3D Gaussians for tomographic reconstruction.
• To achieve high reconstruction quality with minimal training time, outperforming both NeRF-based and traditional CT methods.

Proposed method

• Introduces a novel rectification technique by refactoring the 3D-to-2D projection in 3DGS to correct an integration bias that distorts X-ray attenuation estimation.
• Designs tailored Gaussian kernels optimized for X-ray imaging, improving the fidelity of ray integration along projection paths.
• Extends the 3DGS rasterization process to handle X-ray projection geometry, enabling differentiable rendering of attenuation values through 3D Gaussians.
• Develops a CUDA-based differentiable voxelizer that enables efficient, differentiable volume reconstruction from 3D Gaussian distributions.
• Employs FDK initialization to provide a coarse starting structure, accelerating convergence and improving final reconstruction quality.
• Applies total variation (TV) regularization to suppress artifacts and promote smoothness in the reconstructed density field.

Figure 1: We compare our method to state-of-the-art NeRF-based methods (IntraTomo [ 61 ] , NAF [ 62 ] , SAX-NeRF [ 7 ] ) in terms of visual quality, PSNR (dB), and training time (minute). Our method achieves the highest reconstruction quality and is significantly faster than other methods. \cref@con

Experimental results

Research questions

• RQ1 What integration bias exists in standard 3D Gaussian Splatting when applied to tomographic reconstruction, and how does it affect density field recovery?
• RQ2 Can 3D Gaussian Splatting be effectively adapted for sparse-view X-ray computed tomography with high fidelity and fast convergence?
• RQ3 How do tailored Gaussian kernels and X-ray-specific rasterization improve reconstruction accuracy compared to standard 3DGS?
• RQ4 To what extent does the proposed differentiable voxelizer and optimization strategy enable faster and higher-quality tomographic reconstruction than NeRF-based methods?

Key findings

• R²-Gaussian achieves a PSNR of 38.90 dB on the test set, representing a 0.93 dB improvement over state-of-the-art NeRF-based methods.
• The method reduces training time to under 9 minutes (specifically 8m14s with 50k Gaussians), which is 12× faster than NeRF-based approaches.
• The full model improves PSNR by 1.51 dB and SSIM by 0.018 over the baseline, with FDK initialization contributing 0.9 dB gain.
• Optimal performance is achieved with 50k initialized Gaussians, TV loss weight λ_tv = 0.05, and volume size D = 32.
• Convergence analysis shows R²-Gaussian reaches sharp, artifact-free results by the 500th iteration, significantly outpacing NeRF-based methods.
• The method produces view-independent reconstructions, unlike X-3DGS, which suffers from view-dependent artifacts.

Figure 2: A detection plane captures the attenuation of X-rays emitted from different angles. \cref@constructprefix page \cref@result