[论文解读] NeuSG: Neural Implicit Surface Reconstruction with 3D Gaussian Splatting Guidance

Hanlin Chen, Chen Li+2 more|arXiv (Cornell University)|Dec 1, 2023

3D Shape Modeling and Analysis被引用 14

一句话总结

NeuSG 联合优化 NeuS 与 3D 高斯喷溅（Gaussian Splatting），利用尺度正则化和法线正则化将高斯中心转化为准确的表面先验，实现细致且完备的重建。

ABSTRACT

Existing neural implicit surface reconstruction methods have achieved impressive performance in multi-view 3D reconstruction by leveraging explicit geometry priors such as depth maps or point clouds as regularization. However, the reconstruction results still lack fine details because of the over-smoothed depth map or sparse point cloud. In this work, we propose a neural implicit surface reconstruction pipeline with guidance from 3D Gaussian Splatting to recover highly detailed surfaces. The advantage of 3D Gaussian Splatting is that it can generate dense point clouds with detailed structure. Nonetheless, a naive adoption of 3D Gaussian Splatting can fail since the generated points are the centers of 3D Gaussians that do not necessarily lie on the surface. We thus introduce a scale regularizer to pull the centers close to the surface by enforcing the 3D Gaussians to be extremely thin. Moreover, we propose to refine the point cloud from 3D Gaussians Splatting with the normal priors from the surface predicted by neural implicit models instead of using a fixed set of points as guidance. Consequently, the quality of surface reconstruction improves from the guidance of the more accurate 3D Gaussian splatting. By jointly optimizing the 3D Gaussian Splatting and the neural implicit model, our approach benefits from both representations and generates complete surfaces with intricate details. Experiments on Tanks and Temples verify the effectiveness of our proposed method.

研究动机与目标

• Motivate improved neural implicit surface reconstruction with rich geometric priors beyond traditional depth maps or sparse point clouds.
• Leverage dense, detail-rich 3D Gaussian Splatting to provide strong geometric constraints.
• Develop regularizers to align 3D Gaussian centers with the surface and refine Gaussians via predicted normals.
• Propose a joint optimization framework that benefits from both NeuS and Gaussian Splatting to produce complete, detailed surfaces.

提出的方法

• Use 3D Gaussian Splatting to generate dense point clouds with detailed geometry for surface priors.
• Introduce scale regularization to flatten Gaussians so centers lie near the surface.
• Refine Gaussian priplings by aligning their normals with NeuS-predicted surface normals.
• Jointly optimize NeuS (SDF-based volume rendering) and Gaussian Splatting with coordinated losses.
• Employ a dual-network approach similar to NeRF++ with hash encoding for efficient training.
• Optimize with RGB rendering loss, Eikonal regularization, and point-cloud-based SDF constraints, plus Gaussian-specific losses (scale and normal alignment).

实验结果

研究问题

• RQ1 Can jointly optimizing NeuS with 3D Gaussian Splatting improve surface completeness and detail over existing neural implicit methods?
• RQ2 Do scale regularization and normal alignment sufficiently pull Gaussian centers onto the surface and provide reliable normals for refinement?
• RQ3 How does Gaussian-based regularization compare to traditional point-cloud priors (e.g., from MVS) in guiding SDF-based surface reconstruction?
• RQ4 What are the trade-offs in training efficiency and surface quality when using NeuSG versus baselines on Tanks and Temples scenes?

主要发现

Barn	Caterpillar	Courthouse	Ignatius	Meetingroom	Truck	Mean	GPU hours
0.22	0.18	0.08	0.02	0.08	0.35	0.15	-
0.55	0.01	0.11	0.22	0.19	0.19	0.21	-
0.49	0.21	0.36	0.25	0.43	0.28	0.34	-
0.29	0.29	0.17	0.83	0.24	0.45	0.38	-
0.46	0.32	0.08	0.81	0.08	0.44	0.37	16
0.33	0.26	0.12	0.72	0.20	0.45	0.35	-
0.49	0.31	0.12	0.78	0.23	0.42	0.39	18
0.61	0.34	0.13	0.82	0.22	0.45	0.43	15
0.70	0.36	0.28	0.89	0.32	0.48	0.50	128
0.73	0.37	0.22	0.83	0.35	0.46	0.49	16

• NeuSG achieves the highest surface quality (F1) on Tanks and Temples compared to baselines like NeuS, NeuralWarp, and Geo-Neus.
• NeuSG outperforms NeuS in reconstructing complete surfaces with finer details (0.49 vs 0.38 F1).
• Joint optimization with Gaussian Splatting yields better results than using Gaussian priors alone or MVS priors.
• NeuSG is more efficient than larger-hash NeuralAngelo configurations, achieving comparable results with significantly reduced training time (16 hours vs 128 hours).
• Ablations show that scale and normal regularizations, plus Gaussian priors, substantially improve F1 scores (e.g., 0.73 on mean across scenes).