[Paper Review] Self-Supervised Generative Adversarial Network for Depth Estimation in Laparoscopic Images

Baoru Huang, Jian-Qing Zheng+5 more|arXiv (Cornell University)|Jul 9, 2021

Advanced Vision and Imaging34 references4 citations

TL;DR

This paper proposes SADepth, a self-supervised generative adversarial network for depth estimation in laparoscopic images using stereo pairs as input, leveraging adversarial training and multi-scale supervision to improve reconstruction quality and generalization. It achieves state-of-the-art performance on two public datasets without requiring ground truth depth labels.

ABSTRACT

Dense depth estimation and 3D reconstruction of a surgical scene are crucial steps in computer assisted surgery. Recent work has shown that depth estimation from a stereo images pair could be solved with convolutional neural networks. However, most recent depth estimation models were trained on datasets with per-pixel ground truth. Such data is especially rare for laparoscopic imaging, making it hard to apply supervised depth estimation to real surgical applications. To overcome this limitation, we propose SADepth, a new self-supervised depth estimation method based on Generative Adversarial Networks. It consists of an encoder-decoder generator and a discriminator to incorporate geometry constraints during training. Multi-scale outputs from the generator help to solve the local minima caused by the photometric reprojection loss, while the adversarial learning improves the framework generation quality. Extensive experiments on two public datasets show that SADepth outperforms recent state-of-the-art unsupervised methods by a large margin, and reduces the gap between supervised and unsupervised depth estimation in laparoscopic images.

Motivation & Objective

• To address the lack of per-pixel depth ground truth in laparoscopic imaging, which limits supervised depth estimation.
• To improve unsupervised depth estimation performance by leveraging self-supervised learning with image reconstruction as a supervisory signal.
• To enhance reconstruction quality and reduce artifacts in depth prediction using adversarial training.
• To enable generalization across different laparoscopic systems and datasets without fine-tuning.
• To close the performance gap between supervised and unsupervised depth estimation in endoscopic vision.

Proposed method

• The framework uses a U-Net-based encoder-decoder generator network to predict left and right disparity maps from stereo RGB image pairs.
• A reprojection sampler computes photometric reprojection loss by warping the predicted disparity maps to reconstruct the input images.
• Multi-scale supervision is applied across multiple resolution levels to mitigate local minima and improve feature learning.
• Disparity smoothness loss is incorporated to regularize the predicted disparities and reduce noise.
• A discriminator network is trained to distinguish between real input images and reconstructed images from the generator, enabling adversarial training.
• The overall training objective combines photometric reprojection loss, disparity smoothness loss, and adversarial loss for end-to-end optimization.

Experimental results

Research questions

• RQ1 Can adversarial training improve the quality of reconstructed images in self-supervised depth estimation for laparoscopic images?
• RQ2 How does multi-scale supervision help overcome local minima caused by photometric reprojection loss in stereo depth estimation?
• RQ3 To what extent can a self-supervised GAN-based method generalize across different laparoscopic datasets without ground truth?
• RQ4 Can the proposed method achieve performance competitive with supervised methods despite the absence of per-pixel depth annotations?
• RQ5 How does the integration of adversarial learning affect the fidelity of depth prediction and image reconstruction in low-texture surgical scenes?

Key findings

• SADepth achieved an SSIM of 79.6 on the dVPN dataset, outperforming Monodepth [9] by 24.7 units and Monodepth2 [10] by 8.4 units.
• On the SCARED dataset, SADepth achieved a mean absolute depth error of 17.42 mm on test set 1 and 11.23 mm on test set 2, surpassing all unsupervised baselines.
• The method reduced the performance gap between unsupervised and supervised depth estimation, with results competitive against some supervised approaches on SCARED.
• SADepth generalized well across datasets collected from different laparoscopes and surgical scenarios, demonstrating strong robustness.
• The use of adversarial training significantly improved the visual quality of reconstructed images, leading to better photometric consistency and reduced artifacts.
• The combination of multi-scale supervision and disparity smoothness loss effectively mitigated local minima and improved depth map accuracy.