[Paper Review] Semantically-Guided Representation Learning for Self-Supervised Monocular Depth

Vitor Guizilini, Rui Hou+3 more|arXiv (Cornell University)|Feb 27, 2020

Advanced Vision and ImagingComputer Science107 citations

TL;DR

The paper introduces a semantically guided self-supervised monocular depth framework that uses fixed pretrained semantic features via pixel-adaptive convolutions to improve depth representations, plus a two-stage training to mitigate dynamic-object bias.

ABSTRACT

Self-supervised learning is showing great promise for monocular depth estimation, using geometry as the only source of supervision. Depth networks are indeed capable of learning representations that relate visual appearance to 3D properties by implicitly leveraging category-level patterns. In this work we investigate how to leverage more directly this semantic structure to guide geometric representation learning, while remaining in the self-supervised regime. Instead of using semantic labels and proxy losses in a multi-task approach, we propose a new architecture leveraging fixed pretrained semantic segmentation networks to guide self-supervised representation learning via pixel-adaptive convolutions. Furthermore, we propose a two-stage training process to overcome a common semantic bias on dynamic objects via resampling. Our method improves upon the state of the art for self-supervised monocular depth prediction over all pixels, fine-grained details, and per semantic categories.

Motivation & Objective

• Leverage fixed pretrained semantic segmentation to guide self-supervised monocular depth learning.
• Incorporate semantic-aware feature guidance into the depth network via pixel-adaptive convolutions.
• Address semantic bias on dynamic objects with a two-stage training process to reduce infinite-depth artifacts.
• Demonstrate improvements over state-of-the-art self-supervised monocular depth on KITTI across pixels, classes, and dynamic objects.

Proposed method

• Use a fixed pretrained semantic segmentation network to guide depth features.
• Apply pixel-adaptive convolutions where guidance features modulate convolutional weights via a Gaussian kernel over semantic features.
• Process semantic features through multi-level maps and integrate via guidance equations (Eq. 4 and 5).
• Train depth/pose networks in a self-supervised SfM setup with photometric and edge-aware depth losses.
• Implement a two-stage training pipeline to de-bias infinite-depth predictions on dynamic objects by resampling training data based on an initial depth model.

Experimental results

Research questions

• RQ1 Can fixed semantic guidance improve self-supervised monocular depth representations without semantic supervision on target data?
• RQ2 Do pixel-adaptive convolutions with semantic guidance yield more accurate and boundary-sharp depth maps across semantic classes?
• RQ3 Can a two-stage training scheme mitigate infinite-depth bias on dynamic objects in self-supervised depth estimation?
• RQ4 Is the proposed approach compatible with multiple depth network architectures and scales?
• RQ5 How does class- or pixel-level depth performance improve with semantic guidance compared to baselines?

Key findings

• Outperforms state-of-the-art self-supervised monocular depth methods on KITTI across standard metrics.
• Semantically-guided depth features yield sharper boundaries and better Delineation for distant and fine-grained structures.
• Two-stage training reduces infinite-depth artifacts on dynamic objects, improving class-average Abs-Rel (notably for cars and motorcycles).
• The approach consistently improves depth across different encoder backbones (ResNet-18, ResNet-50, PackNet).
• Ablation shows semantic guidance and two-stage training both contribute to performance gains.