[Paper Review] DESC: Domain Adaptation for Depth Estimation via Semantic Consistency

Adrian Lopez-Rodriguez, Krystian Mikolajczyk|arXiv (Cornell University)|Sep 3, 2020

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes DESC, a domain adaptation method for monocular depth estimation that leverages semantic consistency and instance-level size priors to generate pseudo-labels in an unannotated target domain. By enforcing consistency between a depth model and a secondary model trained on semantic and edge features, DESC achieves state-of-the-art performance on Virtual KITTI to KITTI and Cityscapes to KITTI benchmarks, reducing absolute relative error by up to 9.8%.

ABSTRACT

Accurate real depth annotations are difficult to acquire, needing the use of special devices such as a LiDAR sensor. Self-supervised methods try to overcome this problem by processing video or stereo sequences, which may not always be available. Instead, in this paper, we propose a domain adaptation approach to train a monocular depth estimation model using a fully-annotated source dataset and a non-annotated target dataset. We bridge the domain gap by leveraging semantic predictions and low-level edge features to provide guidance for the target domain. We enforce consistency between the main model and a second model trained with semantic segmentation and edge maps, and introduce priors in the form of instance heights. Our approach is evaluated on standard domain adaptation benchmarks for monocular depth estimation and show consistent improvement upon the state-of-the-art.

Motivation & Objective

• To address the challenge of acquiring accurate depth annotations, which are costly and require specialized sensors like LiDAR.
• To improve monocular depth estimation in a target domain lacking depth annotations by leveraging semantic annotations from a source domain.
• To bridge the domain gap between synthetic (source) and real (target) datasets using semantic consistency and instance-based depth priors.
• To enhance depth estimation performance without requiring stereo or video supervision in the target domain.

Proposed method

• Leverages a pre-trained panoptic segmentation model to extract semantic and edge features from the target domain for guidance.
• Uses instance-level size priors learned from the source domain to generate depth pseudo-labels in the target domain.
• Introduces a consistency loss between the main depth estimation network and a secondary network that predicts depth from semantic and edge maps.
• Employs a learnable scalar to adaptively scale the consistency loss, improving generalization across domain shifts.
• Trains the main model using a combination of self-supervised depth loss and the semantic consistency loss.
• Utilizes a two-branch architecture where the secondary network is trained on semantic and edge inputs to enforce structural consistency with the main model.

Experimental results

Research questions

• RQ1 Can semantic consistency between source and target domains improve depth estimation in a domain adaptation setting with no target depth annotations?
• RQ2 How effective is the use of instance-level size priors as pseudo-labels for depth estimation in the target domain?
• RQ3 Does enforcing consistency between a depth model and a model trained on semantic and edge features improve generalization across domain shifts?
• RQ4 To what extent does the proposed method outperform existing domain adaptation approaches for monocular depth estimation?
• RQ5 Can the method generalize across diverse datasets such as Virtual KITTI to KITTI and Cityscapes to KITTI without requiring additional supervision?

Key findings

• DESC achieves a 9.8% lower absolute relative error (Sq Rel) compared to the state-of-the-art on the Virtual KITTI to KITTI benchmark.
• On the Cityscapes to KITTI benchmark, DESC reduces absolute relative error by 13.9% compared to T 2 Net, demonstrating strong domain generalization.
• The consistency term with the semantic-guided network contributes significantly to performance gains, as shown by ablation studies.
• DESC outperforms Struct2Depth (M+R), which uses three-frame refinement, while requiring only a single image for inference.
• The method achieves state-of-the-art results even without ImageNet pretraining, outperforming Monodepth2 in the same setup.
• Qualitative results show improved geometric completeness of predicted depth maps, especially for objects like cars and poles, compared to T 2 Net and GASDA.