[Paper Review] Spherical Space Feature Decomposition for Guided Depth Map Super-Resolution

Zixiang Zhao, Jiangshe Zhang+6 more|arXiv (Cornell University)|Mar 15, 2023

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes SSDNet, a novel spherical space feature decomposition network for guided depth map super-resolution that leverages Restormer-based encoders to extract cross-modal features, maps them to spherical space for effective separation of domain-shared and domain-private features, and employs a spherical contrast refinement module to address blurry edges, noisy surfaces, and over-transferred RGB texture. The method achieves state-of-the-art performance with fewer parameters and superior generalization on real-world scenes.

ABSTRACT

Guided depth map super-resolution (GDSR), as a hot topic in multi-modal image processing, aims to upsample low-resolution (LR) depth maps with additional information involved in high-resolution (HR) RGB images from the same scene. The critical step of this task is to effectively extract domain-shared and domain-private RGB/depth features. In addition, three detailed issues, namely blurry edges, noisy surfaces, and over-transferred RGB texture, need to be addressed. In this paper, we propose the Spherical Space feature Decomposition Network (SSDNet) to solve the above issues. To better model cross-modality features, Restormer block-based RGB/depth encoders are employed for extracting local-global features. Then, the extracted features are mapped to the spherical space to complete the separation of private features and the alignment of shared features. Shared features of RGB are fused with the depth features to complete the GDSR task. Subsequently, a spherical contrast refinement (SCR) module is proposed to further address the detail issues. Patches that are classified according to imperfect categories are input into the SCR module, where the patch features are pulled closer to the ground truth and pushed away from the corresponding imperfect samples in the spherical feature space via contrastive learning. Extensive experiments demonstrate that our method can achieve state-of-the-art results on four test datasets, as well as successfully generalize to real-world scenes. The code is available at \url{https://github.com/Zhaozixiang1228/GDSR-SSDNet}.

Motivation & Objective

• To address the challenge of effectively extracting and distinguishing domain-shared and domain-private features in RGB and depth modalities for guided depth map super-resolution.
• To mitigate three key detail issues in depth map super-resolution: blurry edges, noisy surfaces, and over-transferred RGB texture.
• To improve cross-modality feature alignment and separation using a spherical feature space, avoiding limitations of Euclidean distance in scale-sensitive feature spaces.
• To develop a lightweight, generalizable model that performs well on both synthetic and real-world depth map datasets without fine-tuning.

Proposed method

• The method employs Restormer blocks as encoders to extract local and global features from low-resolution depth maps and high-resolution RGB images.
• Extracted features are mapped into spherical space to enable robust separation of domain-private features and alignment of domain-shared features.
• A spherical contrast refinement (SCR) module uses contrastive learning to pull patch features closer to ground truth and push them away from imperfect samples in spherical space.
• The model uses a multi-loss training objective including pixel-wise reconstruction loss, feature separation loss, and alignment loss in spherical space.
• The decoder is also based on Restormer blocks to reconstruct high-resolution depth maps from the refined features.
• The entire framework is trained in an end-to-end manner with a focus on preserving depth discontinuities and reducing texture artifacts.

Experimental results

Research questions

• RQ1 Can spherical space feature decomposition improve the modeling of cross-modal dependencies between RGB and depth features in guided depth super-resolution?
• RQ2 How effectively can spherical contrastive learning reduce common artifacts such as blurry edges and over-transferred RGB texture in depth map reconstruction?
• RQ3 Does separating domain-shared and domain-private features in spherical space lead to better generalization on real-world depth map data compared to Euclidean-based methods?
• RQ4 To what extent does the proposed method outperform existing SOTA approaches in terms of both quantitative metrics and perceptual quality across multiple datasets and super-resolution factors?

Key findings

• SSDNet achieves state-of-the-art performance on four benchmark datasets (RGBDD, Middlebury, etc.) across scaling factors of ×4, ×8, and ×16, with significant improvements in PSNR and SSIM.
• The model achieves superior generalization on real-world scenes without fine-tuning, outperforming prior methods in RMSE on the real-world branch of the RGBDD dataset.
• Ablation studies confirm that the spherical feature decomposition and SCR module are critical for performance, with degradation observed when either component is removed.
• The model maintains a lightweight architecture with fewer parameters than competing methods, demonstrating computational efficiency and potential for real-time deployment.
• The spherical space distance measure outperforms ℓ₂ distance in feature separation and alignment, validating its suitability for cross-modal feature learning.
• Visual comparisons show that SSDNet produces sharper edges and cleaner surfaces with reduced texture artifacts, confirming its effectiveness in addressing key detail issues.