[Paper Review] Improving 360 Monocular Depth Estimation via Non-local Dense Prediction Transformer and Joint Supervised and Self-supervised Learning

Ilwi Yun, Hyuk-Jae Lee+1 more|arXiv (Cornell University)|Sep 22, 2021

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes a novel 360° monocular depth estimation method using a non-local dense prediction transformer and joint supervised-self-supervised learning to overcome data scarcity and instability in existing approaches. By leveraging gravity-aligned videos for self-supervised training and combining it with supervised learning, the method achieves state-of-the-art performance on multiple benchmarks, with improved generalization and depth reconstruction accuracy, especially in challenging scenes with occlusions and textureless regions.

ABSTRACT

Due to difficulties in acquiring ground truth depth of equirectangular (360) images, the quality and quantity of equirectangular depth data today is insufficient to represent the various scenes in the world. Therefore, 360 depth estimation studies, which relied solely on supervised learning, are destined to produce unsatisfactory results. Although self-supervised learning methods focusing on equirectangular images (EIs) are introduced, they often have incorrect or non-unique solutions, causing unstable performance. In this paper, we propose 360 monocular depth estimation methods which improve on the areas that limited previous studies. First, we introduce a self-supervised 360 depth learning method that only utilizes gravity-aligned videos, which has the potential to eliminate the needs for depth data during the training procedure. Second, we propose a joint learning scheme realized by combining supervised and self-supervised learning. The weakness of each learning is compensated, thus leading to more accurate depth estimation. Third, we propose a non-local fusion block, which can further retain the global information encoded by vision transformer when reconstructing the depths. With the proposed methods, we successfully apply the transformer to 360 depth estimations, to the best of our knowledge, which has not been tried before. On several benchmarks, our approach achieves significant improvements over previous works and establishes a state of the art.

Motivation & Objective

• Address the scarcity and poor quality of 360° equirectangular depth data, which limits supervised learning performance.
• Overcome the instability and non-unique solutions common in self-supervised 360° depth learning, especially from reflective or textureless surfaces.
• Introduce a joint learning scheme that combines supervised and self-supervised signals to compensate for the weaknesses of each individual approach.
• Develop a non-local fusion block to enhance global feature retention in vision transformers for dense depth prediction.
• Enable effective training of vision transformers on 360° depth estimation despite limited large-scale labeled data.

Proposed method

• Proposes a self-supervised learning method using only gravity-aligned video sequences, eliminating the need for ground-truth depth during training.
• Introduces a joint learning objective combining supervised loss ($\mathcal{L}_{pix} + \mathcal{L}_{grad}$) and self-supervised losses ($\mathcal{L}_{I} + \mathcal{L}_{D}$) to improve robustness and accuracy.
• Designs a non-local fusion block (NLFB) that enhances global context modeling by aggregating long-range dependencies in feature maps during depth reconstruction.
• Utilizes a vision transformer backbone pre-trained on rectilinear images to initialize features, enabling effective fine-tuning on 360° data despite limited supervision.
• Applies a consistency loss ($\mathcal{L}_{D}$) between predicted depth and warped features from neighboring frames to stabilize self-supervised learning.
• Employs a multi-scale supervision strategy with multi-level supervision to improve feature representation and depth detail.

Experimental results

Research questions

• RQ1 Can self-supervised learning on gravity-aligned video sequences eliminate the need for ground-truth depth annotations in 360° monocular depth estimation?
• RQ2 Does combining supervised and self-supervised learning lead to more accurate and stable depth predictions than either method alone?
• RQ3 Can a non-local fusion block improve global context modeling in vision transformers for dense 360° depth prediction?
• RQ4 How does the joint learning scheme mitigate errors caused by inaccurate or missing ground-truth depths in supervised training?
• RQ5 To what extent does the proposed method generalize to unseen scenes and datasets compared to prior state-of-the-art approaches?

Key findings

• The proposed method achieves state-of-the-art performance on multiple benchmarks, including 3D60 and Structure3D, outperforming prior methods in all metrics.
• Joint learning with both supervised and self-supervised losses significantly improves depth estimation accuracy, especially in regions with textureless or occluded surfaces.
• The non-local fusion block (NLFB) enhances depth reconstruction by preserving global context, reducing artifacts and improving object boundary consistency.
• Qualitative results show that the joint learning scheme correctly distinguishes windows from walls and sofas from floors, while baselines fail in such cases.
• Ablation studies confirm that self-supervised losses ($\mathcal{L}_{I}$ and $\mathcal{L}_{D}$) substantially improve performance when combined with supervised losses, particularly in handling ambiguous or noisy ground truth.
• The model trained with the full joint objective and NLFB achieves the best results across all test sets, demonstrating the effectiveness of each component in isolation and in combination.