[Paper Review] SAFENet: Self-Supervised Monocular Depth Estimation with Semantic-Aware Feature Extraction

Jaehoon Choi, Dongki Jung+2 more|arXiv (Cornell University)|Oct 6, 2020

Advanced Vision and Imaging61 references32 citations

TL;DR

SAFENet improves self-supervised monocular depth estimation by integrating semantic-aware depth features through multi-task learning, achieving state-of-the-art KITTI results and better generalization under adverse conditions.

ABSTRACT

Self-supervised monocular depth estimation has emerged as a promising method because it does not require groundtruth depth maps during training. As an alternative for the groundtruth depth map, the photometric loss enables to provide self-supervision on depth prediction by matching the input image frames. However, the photometric loss causes various problems, resulting in less accurate depth values compared with supervised approaches. In this paper, we propose SAFENet that is designed to leverage semantic information to overcome the limitations of the photometric loss. Our key idea is to exploit semantic-aware depth features that integrate the semantic and geometric knowledge. Therefore, we introduce multi-task learning schemes to incorporate semantic-awareness into the representation of depth features. Experiments on KITTI dataset demonstrate that our methods compete or even outperform the state-of-the-art methods. Furthermore, extensive experiments on different datasets show its better generalization ability and robustness to various conditions, such as low-light or adverse weather.

Motivation & Objective

• Motivate improving depth estimation without ground-truth depths by leveraging semantic information.
• Develop semantic-aware depth features via multi-task learning to mitigate photometric loss limitations.
• Design an encoder with task-specific and shared components to reduce interference between depth and segmentation tasks.
• Offer cross-task information pathways that inject semantic affinity into depth features.
• Demonstrate robustness and generalization under low light and adverse weather conditions.

Proposed method

• Propose a multi-task network with a shared encoder and two decoders for depth and segmentation tasks.
• Use semantic-aware depth features by combining depth representations with semantic information through dedicated modules (CPU and APU).
• Incorporate semantic affinity propagation to guide depth features via a semantic affinity matrix learned from segmentation features (APU).
• Apply task-specific residual adapters and SE blocks to allow task-shared and task-specific feature learning without interference.
• Fuse cross-task information through 1x1 convolutions to enable controlled feature sharing between depth and segmentation decoders.
• Train with a self-supervised photometric loss plus a depth-smoothness loss and a semantic segmentation loss, enabling end-to-end optimization.

Experimental results

Research questions

• RQ1 Can semantic-aware feature extraction improve self-supervised monocular depth estimation over photometric loss alone?
• RQ2 How can multi-task learning be structured to minimize task interference while sharing useful representations between depth and semantic segmentation?
• RQ3 Do semantic-aware depth features generalize better under challenging conditions (low light, fog, rain) than depth-only methods?
• RQ4 What is the impact of semantic supervision on depth accuracy on KITTI and cross-dataset generalization (Virtual KITTI, nuScenes)?

Key findings

• SAFENet achieves state-of-the-art results on KITTI in self-supervised monocular depth estimation without ground-truth depths.
• SAFENet with 1024x320 input reaches Abs Rel 0.106, Sq Rel 0.743, RMSE 4.489, and delta<1.25 of 0.884 on KITTI, outperforming several baselines.
• Semantic-aware depth features improve robustness under adverse weather (fog, rain) on vKITTI compared to Monodepth2 with SE modules.
• Across datasets and conditions, SAFENet shows better generalization and maintains depth quality when illumination or weather changes degrade RGB cues.
• Ablation studies indicate that both the CPU (depth-to-semantic feature sharing) and APU (affinity propagation) contribute to gains, with full modular sharing yielding best results.
• Class-wise analysis suggests depth accuracy improvements for most semantic categories (except sky) and improved handling of moving objects.