[Paper Review] BridgeNet: A Joint Learning Network of Depth Map Super-Resolution and Monocular Depth Estimation

Qi Tang, Runmin Cong+5 more|arXiv (Cornell University)|Jul 27, 2021

Advanced Vision and Imaging53 references4 citations

TL;DR

This paper proposes BridgeNet, a joint learning framework that simultaneously performs depth map super-resolution (DSR) and monocular depth estimation (MDE) without additional supervision. By introducing two specialized bridges—High-frequency Attention Bridge (HABdg) for feature-level guidance and Content Guidance Bridge (CGBdg) for reconstruction-level feedback—the model effectively transfers high-frequency and structural information between tasks, achieving state-of-the-art performance on benchmark datasets, especially at large scaling factors.

ABSTRACT

Depth map super-resolution is a task with high practical application requirements in the industry. Existing color-guided depth map super-resolution methods usually necessitate an extra branch to extract high-frequency detail information from RGB image to guide the low-resolution depth map reconstruction. However, because there are still some differences between the two modalities, direct information transmission in the feature dimension or edge map dimension cannot achieve satisfactory result, and may even trigger texture copying in areas where the structures of the RGB-D pair are inconsistent. Inspired by the multi-task learning, we propose a joint learning network of depth map super-resolution (DSR) and monocular depth estimation (MDE) without introducing additional supervision labels. For the interaction of two subnetworks, we adopt a differentiated guidance strategy and design two bridges correspondingly. One is the high-frequency attention bridge (HABdg) designed for the feature encoding process, which learns the high-frequency information of the MDE task to guide the DSR task. The other is the content guidance bridge (CGBdg) designed for the depth map reconstruction process, which provides the content guidance learned from DSR task for MDE task. The entire network architecture is highly portable and can provide a paradigm for associating the DSR and MDE tasks. Extensive experiments on benchmark datasets demonstrate that our method achieves competitive performance. Our code and models are available at https://rmcong.github.io/proj_BridgeNet.html.

Motivation & Objective

• To address the challenge of texture copying and depth bleeding in color-guided depth super-resolution, where RGB and depth modality discrepancies hinder feature transfer.
• To improve depth map super-resolution performance by leveraging monocular depth estimation as an auxiliary task without requiring additional supervision.
• To design a portable, multi-task learning framework that enables bidirectional knowledge transfer between DSR and MDE networks.
• To overcome the limitations of existing methods that rely on direct feature or edge map transfer, which often misalign modality-specific structures.
• To establish a new paradigm for joint learning between depth super-resolution and monocular depth estimation through structured, task-specific interaction modules.

Proposed method

• The framework jointly trains a depth map super-resolution network (DSRNet) and a monocular depth estimation network (MDENet) in an end-to-end manner.
• The High-frequency Attention Bridge (HABdg) selectively transfers high-frequency features from the MDENet encoder to guide the DSRNet encoder, enhancing edge and texture detail recovery.
• The Content Guidance Bridge (CGBdg) provides content-aware features from the DSRNet decoder to the MDENet decoder, improving depth estimation accuracy through reconstructed depth context.
• A differentiated guidance strategy ensures modality-specific information is transferred appropriately, avoiding misalignment between RGB and depth structures.
• The model uses a multi-task loss function combining L1 and perceptual losses to optimize both tasks simultaneously.
• The architecture is highly modular and portable, enabling easy integration into other depth-related tasks.

Experimental results

Research questions

• RQ1 Can joint learning between depth map super-resolution and monocular depth estimation improve performance without additional supervision?
• RQ2 How can high-frequency details from RGB images be effectively transferred to depth super-resolution while avoiding texture copying?
• RQ3 What is the optimal way to structure cross-task feature interaction to preserve modality-specific structural consistency?
• RQ4 Does using monocular depth estimation as an auxiliary task enhance depth map super-resolution performance, especially at high scaling factors?
• RQ5 How do the proposed HABdg and CGBdg components compare to direct feature concatenation or edge-based guidance in terms of performance and robustness?

Key findings

• BridgeNet achieves the best performance on the Middlebury 2005 dataset at ×8 and ×16 scaling factors, with a MAD of 0.343, outperforming all baselines.
• The ablation study confirms that both HABdg and CGBdg are essential: removing either degrades performance, with HABdg contributing significantly to high-frequency detail recovery.
• The HABdg outperforms simple feature concatenation by 0.033 in MAD (0.343 vs. 0.376), demonstrating its effectiveness in selective, attention-based feature transfer.
• Visual comparisons show that BridgeNet produces sharper boundaries and more accurate depth values than DSRNet alone, especially in complex texture regions.
• The model achieves competitive results on the NYUv2 dataset, confirming generalization across diverse real-world scenes.
• The joint training framework enables mutual improvement: DSRNet benefits from MDE’s high-frequency features, and MDENet gains from DSRNet’s content-aware reconstruction.