[Paper Review] VoGE: A Differentiable Volume Renderer using Gaussian Ellipsoids for Analysis-by-Synthesis

Angtian Wang, Peng Wang+3 more|arXiv (Cornell University)|May 30, 2022

Computer Graphics and Visualization Techniques4 citations

TL;DR

VoGE proposes a differentiable volume renderer using Gaussian ellipsoids as 3D geometric primitives, replacing rasterization with ray tracing to model volume density distributions for accurate occlusion reasoning. It achieves state-of-the-art performance in pose estimation, shape fitting, and texture fitting with real-time inference via CUDA-optimized approximate integration and coarse-to-fine rendering.

ABSTRACT

The Gaussian reconstruction kernels have been proposed by Westover (1990) and studied by the computer graphics community back in the 90s, which gives an alternative representation of object 3D geometry from meshes and point clouds. On the other hand, current state-of-the-art (SoTA) differentiable renderers, Liu et al. (2019), use rasterization to collect triangles or points on each image pixel and blend them based on the viewing distance. In this paper, we propose VoGE, which utilizes the volumetric Gaussian reconstruction kernels as geometric primitives. The VoGE rendering pipeline uses ray tracing to capture the nearest primitives and blends them as mixtures based on their volume density distributions along the rays. To efficiently render via VoGE, we propose an approximate closeform solution for the volume density aggregation and a coarse-to-fine rendering strategy. Finally, we provide a CUDA implementation of VoGE, which enables real-time level rendering with a competitive rendering speed in comparison to PyTorch3D. Quantitative and qualitative experiment results show VoGE outperforms SoTA counterparts when applied to various vision tasks, e.g., object pose estimation, shape/texture fitting, and occlusion reasoning. The VoGE library and demos are available at: https://github.com/Angtian/VoGE.

Motivation & Objective

• Address the limitations of rasterization-based differentiable renderers, which fail to handle overlapping primitives and block gradients to occluded objects.
• Improve occlusion reasoning in differentiable rendering by modeling volume density distributions instead of relying on viewing distance for blending.
• Enable interpretable and modifiable 3D representations using explicit Gaussian ellipsoids, combining the benefits of splatting and volume rendering.
• Develop a real-time, differentiable rendering pipeline that supports end-to-end optimization for 3D vision tasks like pose estimation and shape fitting.

Proposed method

• Represent 3D objects using Gaussian ellipsoids defined by center position and spatial covariance matrix, enabling smooth, differentiable geometry.
• Use ray tracing to compute per-ray volume density functions for each Gaussian ellipsoid, replacing rasterization with continuous density aggregation.
• Approximate the integral of volume density along rays via a closed-form solution to avoid expensive numerical integration.
• Apply a coarse-to-fine rendering strategy to accelerate convergence and improve efficiency during optimization.
• Reconstruct image colors by blending attributes of Gaussian kernels weighted by their reweighted contribution based on density and occupancy.
• Implement a CUDA-optimized version of VoGE to achieve real-time rendering performance comparable to PyTorch3D.

Experimental results

Research questions

• RQ1 Can a differentiable renderer based on volumetric Gaussian ellipsoids outperform rasterization-based methods in occlusion reasoning and gradient flow?
• RQ2 Does modeling volume density distributions instead of viewing distance improve blending accuracy in overlapping or partially occluded scenes?
• RQ3 Can Gaussian ellipsoids serve as effective, differentiable 3D primitives for inverse rendering tasks like shape and texture fitting?
• RQ4 To what extent does the proposed approximate close-form solution maintain rendering quality while enabling real-time performance?

Key findings

• VoGE achieves 69.2% accuracy on the PASCAL3D+ in-wild object pose estimation benchmark, surpassing PyTorch3D (61.0%) and NeMo+DSS (27.8%) under the same setup.
• In shape fitting, VoGE reduces median error to 6.9° (vs. 8.8° for PyTorch3D) and achieves 47.9% accuracy at π/6 threshold, outperforming all baselines.
• The ablation study confirms that blocking gradients to either the occupancy function $ T(l_k) $ or the exponential term $ e^{q_k} $ significantly degrades performance.
• VoGE demonstrates strong generalization in texture extraction, generating plausible novel-view renderings even for out-of-distribution cases like boats and buses.
• The loss curves in shape fitting show that VoGE maintains tight geometric constraints even without explicit normal consistency loss, indicating robust gradient flow.
• VoGE's CUDA implementation enables real-time rendering with performance competitive to PyTorch3D, making it suitable for end-to-end optimization in vision pipelines.