[Paper Review] Stereo Endoscopic Image Super-Resolution Using Disparity-Constrained Parallel Attention

Tianyi Zhang, Yun Gu+3 more|arXiv (Cornell University)|Mar 19, 2020

Advanced Image Processing Techniques9 references4 citations

TL;DR

This paper proposes DCSSRnet, a stereo endoscopic image super-resolution network that leverages disparity-constrained parallel attention to preserve stereo correspondence during super-resolution. By using an atrous parallax-attention module and enforcing consistency between low- and high-resolution disparity maps, the method achieves state-of-the-art performance on laparoscopic image datasets, outperforming existing methods in both PSNR and SSIM metrics.

ABSTRACT

With the popularity of stereo cameras in computer assisted surgery techniques, a second viewpoint would provide additional information in surgery. However, how to effectively access and use stereo information for the super-resolution (SR) purpose is often a challenge. In this paper, we propose a disparity-constrained stereo super-resolution network (DCSSRnet) to simultaneously compute a super-resolved image in a stereo image pair. In particular, we incorporate a disparity-based constraint mechanism into the generation of SR images in a deep neural network framework with an additional atrous parallax-attention modules. Experiment results on laparoscopic images demonstrate that the proposed framework outperforms current SR methods on both quantitative and qualitative evaluations. Our DCSSRnet provides a promising solution on enhancing spatial resolution of stereo image pairs, which will be extremely beneficial for the endoscopic surgery.

Motivation & Objective

• To address the challenge of preserving stereo correspondence during super-resolution of endoscopic image pairs.
• To improve super-resolution quality by incorporating disparity constraints between low- and high-resolution stereo images.
• To develop a symmetric deep learning framework that jointly reconstructs left and right super-resolved images.
• To enhance feature representation using residual atrous spatial pyramid pooling and attention mechanisms aligned with epipolar geometry.
• To achieve more realistic and detailed super-resolved images that maintain structural consistency across stereo views.

Proposed method

• The method employs a symmetric U-Net-like architecture to simultaneously generate super-resolved left and right images from a stereo pair.
• A residual atrous spatial pyramid pooling (ASPP) module is used to extract multi-scale features with large receptive fields.
• An atrous parallax-attention module (APAM) generates disparity masks by aggregating features across multiple lines near the epipolar line, improving alignment robustness.
• A disparity-constraint loss is introduced, enforcing consistency between the disparity maps of low- and high-resolution image pairs.
• The loss function combines reconstruction loss and a regularization term based on the difference between LR and HR disparity masks.
• Trilinear interpolation is used to upsample low-resolution disparity maps for consistent supervision during training.

Experimental results

Research questions

• RQ1 Can disparity constraints improve the quality of super-resolved stereo endoscopic images?
• RQ2 How does enforcing consistency between low- and high-resolution disparity maps affect structural fidelity in super-resolution?
• RQ3 Can a parallel, symmetric network architecture effectively leverage stereo correspondence for enhanced image reconstruction?
• RQ4 Does using multi-line feature aggregation instead of single epipolar lines improve attention-based feature alignment in stereo SR?
• RQ5 To what extent does the proposed method outperform existing single-image and stereo-based super-resolution methods on laparoscopic data?

Key findings

• The proposed DCSSRnet achieves a PSNR of 43.610 dB and SSIM of 0.9957 at a 2× scale, outperforming all compared methods including PASSR and StereoSR.
• At a 4× scale, DCSSRnet achieves a PSNR of 35.598 dB and SSIM of 0.9809, demonstrating superior performance in high-magnification super-resolution.
• Qualitative results show that DCSSRnet produces sharper edges and more realistic texture details compared to bicubic, SRCNN, VDSR, DRRN, and PASSR.
• The disparity-constraint mechanism effectively preserves stereo correspondence, reducing artifacts and improving structural consistency in reconstructed images.
• The method shows limitations when left and right images are inconsistently registered, indicating sensitivity to stereo misalignment.
• The ablation study confirms that the disparity-constraint loss significantly improves performance, validating its role in maintaining stereo fidelity.