[Paper Review] Segment Any 4D Gaussians

Shengxiang Ji, Guanjun Wu+6 more|arXiv (Cornell University)|Jul 5, 2024

Computational Physics and Python Applications5 citations

TL;DR

SA4D extends Segment Anything to 4D Gaussian representations by learning a temporal identity field to address Gaussian drifting, enabling fast, open-world segmentation and dynamic scene editing in 4D Gaussian Splatting.

ABSTRACT

Modeling, understanding, and reconstructing the real world are crucial in XR/VR. Recently, 3D Gaussian Splatting (3D-GS) methods have shown remarkable success in modeling and understanding 3D scenes. Similarly, various 4D representations have demonstrated the ability to capture the dynamics of the 4D world. However, there is a dearth of research focusing on segmentation within 4D representations. In this paper, we propose Segment Any 4D Gaussians (SA4D), one of the first frameworks to segment anything in the 4D digital world based on 4D Gaussians. In SA4D, an efficient temporal identity feature field is introduced to handle Gaussian drifting, with the potential to learn precise identity features from noisy and sparse input. Additionally, a 4D segmentation refinement process is proposed to remove artifacts. Our SA4D achieves precise, high-quality segmentation within seconds in 4D Gaussians and shows the ability to remove, recolor, compose, and render high-quality anything masks. More demos are available at: https://jsxzs.github.io/sa4d/.

Motivation & Objective

• Reformulate 4D segmentation for deformation-based 4D Gaussian representations.
• Develop a temporal identity feature field to address Gaussian drifting across time.
• Integrate a Gaussian identity table and post-processing to refine segmentation quality.
• Leverage 2D supervision from video trackers to train 4D segmentation without GT 4D labels.
• Demonstrate real-time rendering and editing capabilities (removal, recoloring, composition) in 4D scenes.

Proposed method

• Adopt 4D Gaussian Splatting (4D-GS) as the 4D representation with a global canonical 3D Gaussian base and a deformation field.
• Introduce a temporal identity feature field network that predicts time-variant identity features e for each Gaussian from its canonical position and time.
• Use a tiny convolutional decoder and softmax to classify per-Gaussian identity, enabling 2D identity supervision from video tracker masks.
• Define an export process for 4D Gaussians that fuses deformation-based and identity-based predictions to export per-timestamp Gaussians.
• Train with 2D pseudo-segmentation losses (L2D) and 3D regularization losses (L3D) to supervise identity features in the absence of GT 4D labels.
• Apply a 2D segmentation refinement post-processing step to remove outliers and resolve boundary ambiguities, and maintain a Gaussian Identity Table (M) for near-neighbor timestamp interpolation.

Experimental results

Research questions

• RQ1 How can SAM-style segmentation be extended to open-world 4D Gaussian representations?
• RQ2 Can a temporal identity feature field mitigate Gaussian drifting across time in 4D-GS?
• RQ3 What supervision strategy enables 4D segmentation without ground-truth 4D labels?
• RQ4 How can refinement and identity-table mechanisms improve segmentation quality and rendering speed for 4D scenes?
• RQ5 What editing capabilities (removal, recoloring, composition) become feasible with SA4D in dynamic scenes?

Key findings

Model	mIoU (%) (HyperNeRF)	mAcc (%) (HyperNeRF)	mIoU (%) (Neu3D)	mAcc (%) (Neu3D)
SAGA	65.25	75.56	76.26	81.56
Gaussian Grouping	69.53	91.55	87.02	98.72
Ours w/o TFF (w/o Refinement)	80.26	99.56	-	-
Ours w/ TFF (w/o Refinement)	81.10	99.54	80.14	99.88
Ours w/ all	89.86	99.24	93.02	99.76

• SA4D achieves fast interactive 4D segmentation within seconds on an RTX 3090.
• Incorporating a temporal identity field reduces Gaussian drifting and improves ID consistency across time.
• Temporal identity supervision from 2D video tracker masks plus 3D regularization yields high segmentation accuracy on dynamic scenes compared to 3D baselines.
• Gaussian Identity Table enables near-real-time rendering and editing with negligible extra storage compared to baseline 4D-GS.
• Refinement steps significantly reduce artifacts and boundary ambiguities, improving IoU and accuracy on dynamic scenes.