[Paper Review] DDGS-CT: Direction-Disentangled Gaussian Splatting for Realistic Volume Rendering

Zhongpai Gao, Benjamin Planche+4 more|arXiv (Cornell University)|Jun 4, 2024

Computer Graphics and Visualization Techniques4 citations

TL;DR

This paper proposes DDGS-CT, a direction-disentangled 3D Gaussian splatting method that separates isotropic and direction-dependent radiosity components to enable efficient, differentiable, and realistic volume rendering of digitally reconstructed radiographs (DRRs) from CT volumes. By modeling Compton scattering and other anisotropic X-ray effects without runtime Monte Carlo simulations, DDGS achieves state-of-the-art image accuracy and superior performance in 2D/3D CT registration, enabling real-time intraoperative applications.

ABSTRACT

Digitally reconstructed radiographs (DRRs) are simulated 2D X-ray images generated from 3D CT volumes, widely used in preoperative settings but limited in intraoperative applications due to computational bottlenecks, especially for accurate but heavy physics-based Monte Carlo methods. While analytical DRR renderers offer greater efficiency, they overlook anisotropic X-ray image formation phenomena, such as Compton scattering. We present a novel approach that marries realistic physics-inspired X-ray simulation with efficient, differentiable DRR generation using 3D Gaussian splatting (3DGS). Our direction-disentangled 3DGS (DDGS) method separates the radiosity contribution into isotropic and direction-dependent components, approximating complex anisotropic interactions without intricate runtime simulations. Additionally, we adapt the 3DGS initialization to account for tomography data properties, enhancing accuracy and efficiency. Our method outperforms state-of-the-art techniques in image accuracy. Furthermore, our DDGS shows promise for intraoperative applications and inverse problems such as pose registration, delivering superior registration accuracy and runtime performance compared to analytical DRR methods.

Motivation & Objective

• To address the computational bottleneck of physics-based Monte Carlo DRR rendering in real-time intraoperative applications.
• To improve image realism in DRR generation by modeling anisotropic X-ray effects such as Compton scattering, which analytical methods overlook.
• To enable differentiable, efficient DRR rendering using 3D Gaussian splatting while preserving physical accuracy.
• To enhance 3DGS initialization for tomographic data by incorporating geometry and radiodensity-aware sampling.
• To demonstrate superior performance in downstream inverse problems like 2D/3D CT image registration.

Proposed method

• Introduce a direction-disentangled 3D Gaussian splatting (DDGS) framework that decomposes radiosity into non-directional (isotropic) and direction-dependent (anisotropic) components.
• Model direction-dependent effects using a learnable function that implicitly captures scattering without explicit photon transport simulation.
• Adapt 3DGS initialization with a dual strategy: sampling contact regions between anatomical structures and homogeneous regions for improved accuracy and efficiency.
• Use a differentiable rendering pipeline that supports backpropagation for inverse problems such as pose registration.
• Leverage a pre-trained ground-truth DRR renderer (e.g., Monte Carlo) to supervise DDGS training, ensuring physical realism.
• Optimize the feature dimensionality and ablate component disentanglement to validate design choices.

Figure 1: Proposed pipeline of Direction-Disentangled 3D Gaussian Splatting (DDGS).

Experimental results

Research questions

• RQ1 Can a direction-disentangled 3D Gaussian splatting approach achieve realistic DRR rendering without relying on expensive Monte Carlo simulations?
• RQ2 How does separating isotropic and anisotropic radiosity components improve image fidelity compared to standard 3DGS or analytical DRR methods?
• RQ3 To what extent does the proposed radiodensity-aware initialization enhance rendering accuracy and convergence in medical volume rendering?
• RQ4 Can DDGS outperform analytical DRR methods in both image quality and runtime performance for 2D/3D CT registration?
• RQ5 Does the direction-dependent component effectively model Compton scattering and other anisotropic X-ray effects in clinical imaging scenarios?

Key findings

• DDGS achieves a mean PSNR of 7.46 and SSIM of 0.80 on the CTPelvic1K dataset, significantly outperforming DiffDRR (PSNR: 30.88, SSIM: 0.72) and X-Gaussian in image quality.
• The ablation study confirms that direction disentanglement is essential: entangled or single-component models degrade performance, with 3DGS-disentangled baseline underperforming DDGS.
• The proposed radiodensity-aware initialization outperforms both uniform and even sampling baselines, improving PSNR and reducing registration error.
• Increasing feature dimension improves performance, but gains plateau beyond k=16, indicating diminishing returns.
• In 2D/3D registration, DDGS achieves the lowest rotation (0.80°) and translation (0.65 mm) errors on CTPelvic1K, surpassing analytical methods.
• On the Ljubljana dataset, DDGS enables faster convergence in pose refinement compared to DiffDRR, demonstrating superior runtime efficiency for inverse problems.

Figure 2: Illustration of the different sampling strategies for 3DGS initialization.