[Paper Review] 3D endoscopic depth estimation using 3D surface-aware constraints

Shang Zhao, Ce Wang+3 more|arXiv (Cornell University)|Mar 4, 2022

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes a novel 3D surface-aware loss (Oriented Point loss) for endoscopic depth estimation that leverages 3D geometric constraints from camera parameters to improve depth accuracy and convergence. By integrating a specularity removal module and surface-aware supervision, the method achieves state-of-the-art performance on SCARED and Hamlyn datasets, reducing MAE by 48.62mm and RMSE by 43.46mm over baseline methods.

ABSTRACT

Robotic-assisted surgery allows surgeons to conduct precise surgical operations with stereo vision and flexible motor control. However, the lack of 3D spatial perception limits situational awareness during procedures and hinders mastering surgical skills in the narrow abdominal space. Depth estimation, as a representative perception task, is typically defined as an image reconstruction problem. In this work, we show that depth estimation can be reformed from a 3D surface perspective. We propose a loss function for depth estimation that integrates the surface-aware constraints, leading to a faster and better convergence with the valid information from spatial information. In addition, camera parameters are incorporated into the training pipeline to increase the control and transparency of the depth estimation. We also integrate a specularity removal module to recover more buried image information. Quantitative experimental results on endoscopic datasets and user studies with medical professionals demonstrate the effectiveness of our method.

Motivation & Objective

• To address the lack of 3D spatial perception in robotic-assisted endoscopic surgery, which limits situational awareness and surgical precision.
• To improve depth estimation in endoscopic images by leveraging 3D geometric constraints beyond 2D pixel-level supervision.
• To recover texture information buried under specular highlights through a dedicated specularity removal module.
• To enhance training stability and convergence by incorporating camera parameters and surface-aware geometric priors into the loss function.
• To validate the method’s clinical relevance through quantitative evaluation and user studies with experienced surgeons.

Proposed method

• Introduces an Oriented Point (OP) loss that encodes 3D surface properties using back-projected depth pixels and camera parameters to form 3D oriented points.
• The OP loss combines two components: point distance for boundary accuracy and normal similarity for local surface smoothness.
• Incorporates camera parameters into the training pipeline to enable explicit geometric control and improve transparency of depth estimation.
• Employs a plug-in specularity removal module to restore texture features in specular regions, enhancing feature extraction for stereo matching.
• Uses a modular pipeline: specularity removal → CNN feature extraction → stereo matching → OP loss supervision for depth prediction.
• Employs back-projection of depth pixels using intrinsic and extrinsic camera parameters to compute 3D surface points for loss computation.

Experimental results

Research questions

• RQ1 Can 3D surface-aware geometric constraints improve depth estimation performance in endoscopic images?
• RQ2 How does incorporating camera parameters into the training pipeline affect the transparency and control of depth estimation?
• RQ3 To what extent does specularity removal enhance depth estimation accuracy in endoscopic images with high reflectivity?
• RQ4 Does the proposed Oriented Point loss lead to faster convergence and better generalization than standard 2D image-space losses?
• RQ5 How does the model's depth prediction compare to ground truth and human perception in clinical settings?

Key findings

• The proposed OP loss outperforms baseline methods by reducing MAE by 48.62mm and RMSE by 43.46mm on the full SCARED test set.
• On keyframe 3 of the SCARED dataset, the model with OP loss achieves 6.16mm lower MAE and 4.44mm lower RMSE compared to the baseline with no specularity removal.
• The user study shows that surgeons rated the proposed method’s depth predictions as having higher spatial correlation with RGB images (mean score 3.7 vs. 3.0 for ground truth) and better 3D visual quality.
• The specularity removal module improves performance by 3.79mm in MAE and 1.05mm in RMSE on the full SCARED test set, confirming its effectiveness.
• The OP loss with both point distance and normal similarity components achieves the best overall performance, outperforming individual components on RMSE, indicating joint contribution to robustness.
• Qualitative results show that the OP loss recovers fine-grained surface details, such as curved boundaries, that are missed by baseline models and even partially absent in ground truth.