[Paper Review] SC-DepthV3: Robust Self-supervised Monocular Depth Estimation for Dynamic Scenes

Libo Sun, Jia-Wang Bian+4 more|arXiv (Cornell University)|Nov 7, 2022

Advanced Vision and Imaging4 citations

TL;DR

SC-DepthV3 proposes a robust self-supervised monocular depth estimation method for dynamic scenes by leveraging pseudo-depth from a pretrained single-image depth model. It introduces Dynamic Region Refinement (DRR) and Local Structure Refinement (LSR) modules that use depth ordinal consistency and normal-guided ranking to improve accuracy and sharpness at object boundaries, significantly outperforming prior self-supervised methods on six benchmark datasets, including challenging dynamic scenes.

ABSTRACT

Self-supervised monocular depth estimation has shown impressive results in static scenes. It relies on the multi-view consistency assumption for training networks, however, that is violated in dynamic object regions and occlusions. Consequently, existing methods show poor accuracy in dynamic scenes, and the estimated depth map is blurred at object boundaries because they are usually occluded in other training views. In this paper, we propose SC-DepthV3 for addressing the challenges. Specifically, we introduce an external pretrained monocular depth estimation model for generating single-image depth prior, namely pseudo-depth, based on which we propose novel losses to boost self-supervised training. As a result, our model can predict sharp and accurate depth maps, even when training from monocular videos of highly-dynamic scenes. We demonstrate the significantly superior performance of our method over previous methods on six challenging datasets, and we provide detailed ablation studies for the proposed terms. Source code and data will be released at https://github.com/JiawangBian/sc_depth_pl

Motivation & Objective

• To address the degradation in self-supervised monocular depth estimation caused by dynamic objects and occlusions in video training.
• To improve depth map sharpness at object boundaries where multi-view consistency fails due to occlusions.
• To leverage imperfect but ordinal-accurate pseudo-depth from a pretrained model to guide self-supervised learning without ground-truth labels.
• To develop a method that maintains high performance on dynamic regions without requiring explicit dynamic object detection or complex motion modeling.
• To demonstrate that off-the-shelf pseudo-depth can be effectively used to boost self-supervised depth estimation with minimal additional cost.

Proposed method

• Uses a pretrained monocular depth model (LeReS) to generate pseudo-depth as a single-image depth prior for self-supervised training.
• Introduces Dynamic Region Refinement (DRR) to enforce depth ordinal consistency between dynamic and static regions via confidence-aware sampling and depth ranking loss.
• Proposes Local Structure Refinement (LSR) to preserve local geometry using normal matching loss and edge-guided relative normal ranking loss.
• Combines photometric and geometry consistency losses with the new pseudo-depth-based regularization to improve training stability and accuracy.
• Employs a multi-stage training strategy that integrates pseudo-depth supervision while maintaining the self-supervised objective.
• Uses off-the-shelf pseudo-depth inference, avoiding retraining or fine-tuning of the external model, thus minimizing computational overhead.

Experimental results

Research questions

• RQ1 Can pseudo-depth from a pretrained single-image depth model improve self-supervised depth estimation in dynamic scenes?
• RQ2 How can imperfect pseudo-depth with low quantitative accuracy still be effectively used to enhance self-supervised depth learning?
• RQ3 Can depth ordinal consistency between dynamic and static regions be leveraged to improve generalization and boundary sharpness?
• RQ4 Does incorporating normal information via edge-guided losses lead to better local structure preservation in depth maps?
• RQ5 Can the proposed method outperform state-of-the-art self-supervised methods on both static and highly dynamic benchmark datasets?

Key findings

• SC-DepthV3 achieves state-of-the-art performance on six challenging datasets, including dynamic benchmarks like DDAD and IBims-1, with a 0.142 AbsRel on DDAD using LeReS(ResNet-101) pseudo-depth.
• The ablation study confirms that removing the DRR or LSR modules leads to significant performance degradation, proving their effectiveness.
• The method outperforms previous self-supervised methods by a large margin, especially in dynamic regions and at object boundaries, as shown in qualitative results in Fig. 1 and Fig. 5(d).
• Using LeReS(ResNet-101) for pseudo-depth generation yields better results than DPT or LeReS(ResNet-50), indicating the importance of normal information alignment.
• The performance gain is consistent across both dynamic and static scenes, demonstrating robustness to scene complexity.
• Even with low-accuracy pseudo-depth, the proposed method achieves high-accuracy depth estimation, proving that ordinal consistency is sufficient for effective supervision.