[Paper Review] PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics

Tianyi Xie, Zeshun Zong+5 more|arXiv (Cornell University)|Nov 20, 2023

Computer Graphics and Visualization Techniques4 citations

TL;DR

PhysGaussian introduces a unified simulation-rendering pipeline that integrates Newtonian physics into 3D Gaussian kernels using a custom Material Point Method (MPM), enabling high-fidelity, physically grounded novel motion synthesis. By evolving kinematic and mechanical attributes—such as velocity, strain, stress, and elastic energy—within a continuum mechanics framework, the method achieves real-time, artifact-free dynamics across diverse materials including elastic solids, metals, non-Newtonian fluids, and granular media.

ABSTRACT

We introduce PhysGaussian, a new method that seamlessly integrates physically grounded Newtonian dynamics within 3D Gaussians to achieve high-quality novel motion synthesis. Employing a custom Material Point Method (MPM), our approach enriches 3D Gaussian kernels with physically meaningful kinematic deformation and mechanical stress attributes, all evolved in line with continuum mechanics principles. A defining characteristic of our method is the seamless integration between physical simulation and visual rendering: both components utilize the same 3D Gaussian kernels as their discrete representations. This negates the necessity for triangle/tetrahedron meshing, marching cubes, "cage meshes," or any other geometry embedding, highlighting the principle of "what you see is what you simulate (WS$^2$)." Our method demonstrates exceptional versatility across a wide variety of materials--including elastic entities, metals, non-Newtonian fluids, and granular materials--showcasing its strong capabilities in creating diverse visual content with novel viewpoints and movements. Our project page is at: https://xpandora.github.io/PhysGaussian/

Motivation & Objective

• To close the gap between physical simulation and visual rendering by unifying the representation of materials for both simulation and rendering.
• To eliminate the need for intermediate geometry representations such as meshes, tetrahedra, or cage meshes in physics-based animation.
• To enable realistic, dynamic behavior—including inertia, volume preservation, and plasticity—within a 3D Gaussian-based framework.
• To achieve real-time performance for complex material dynamics while maintaining high visual fidelity and physical accuracy.
• To support flexible user control over material properties such as Young’s modulus and Poisson’s ratio through internal filling and physical parameterization.

Proposed method

• The method extends 3D Gaussians with physical attributes including velocity, strain, stress, elastic energy, and plasticity, evolved via a PDE-driven displacement field grounded in continuum mechanics.
• A custom Material Point Method (MPM) is employed to simulate the physical evolution of 3D Gaussians, ensuring consistency between simulation and rendering.
• The same 3D Gaussian representation is used for both simulation and rendering, enforcing the 'what you see is what you simulate' (WS²) principle and eliminating resolution mismatches.
• Anisotropy regularization is introduced via a loss function (Eq. 12) to suppress visual artifacts from overly slender Gaussian ellipsoids during large deformations.
• Internal filling is implemented using a reconstructed density field derived from Gaussian opacity, enabling realistic internal dynamics and volume preservation.
• Spherical harmonics orientations are dynamically rotated during deformation, improving visual consistency and reducing artifacts compared to fixed or rigidly transformed harmonics.

Experimental results

Research questions

• RQ1 Can 3D Gaussians be extended with physical kinematics and mechanical properties to enable realistic, dynamic simulation without meshing?
• RQ2 How can a unified 3D Gaussian representation simultaneously support high-fidelity rendering and physically accurate simulation?
• RQ3 To what extent can volume conservation and material-specific behaviors (e.g., elasticity, plasticity) be preserved in a Gaussians-based simulation framework?
• RQ4 How does dynamic rotation of spherical harmonics and anisotropy regularization affect visual fidelity during large deformations?
• RQ5 Can internal filling of 3D Gaussians improve the realism of physical dynamics in hollow, shell-like representations?

Key findings

• PhysGaussian achieves state-of-the-art performance on benchmark dynamic NeRF editing tasks, with a PSNR of 26.96, SSIM of 26.46, and LPIPS of 31.15 on the 'Twist' sequence.
• The ablation study confirms that all components—kinematic deformation, rotating spherical harmonics, and dynamic covariance—are essential, as removing any leads to significant visual artifacts and performance degradation.
• Internal filling enables realistic collapse resistance under gravity, preventing structural collapse seen in hollow Gaussian models.
• Volume conservation is accurately maintained, outperforming geometry-based methods like NeRF-Editing [48], which fail to preserve volume during deformation.
• Anisotropy regularization effectively suppresses burr-like visual artifacts in slender Gaussians during large deformations, improving visual fidelity.
• The method enables real-time performance for scenes with simple dynamics, combining efficient MPM simulation with real-time 3D Gaussian splatting rendering.