[Paper Review] Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction

Huangying Zhan, Ravi Garg+4 more|arXiv (Cornell University)|Mar 11, 2018

Advanced Vision and ImagingComputer Science33 references85 citations

TL;DR

The paper presents an unsupervised framework that jointly learns single-view depth and monocular visual odometry from stereo video sequences, using both spatial and temporal photometric losses and a deep feature reconstruction loss to achieve metric-scale depth and competitive VO.

ABSTRACT

Despite learning based methods showing promising results in single view depth estimation and visual odometry, most existing approaches treat the tasks in a supervised manner. Recent approaches to single view depth estimation explore the possibility of learning without full supervision via minimizing photometric error. In this paper, we explore the use of stereo sequences for learning depth and visual odometry. The use of stereo sequences enables the use of both spatial (between left-right pairs) and temporal (forward backward) photometric warp error, and constrains the scene depth and camera motion to be in a common, real-world scale. At test time our framework is able to estimate single view depth and two-view odometry from a monocular sequence. We also show how we can improve on a standard photometric warp loss by considering a warp of deep features. We show through extensive experiments that: (i) jointly training for single view depth and visual odometry improves depth prediction because of the additional constraint imposed on depths and achieves competitive results for visual odometry; (ii) deep feature-based warping loss improves upon simple photometric warp loss for both single view depth estimation and visual odometry. Our method outperforms existing learning based methods on the KITTI driving dataset in both tasks. The source code is available at https://github.com/Huangying-Zhan/Depth-VO-Feat

Motivation & Objective

• Motivate and address the scale ambiguity in monocular depth and pose estimation by leveraging stereo training data.
• Jointly learn a depth estimator and a visual odometry network to enforce cross-task consistency.
• Improve supervision beyond photometric loss by introducing a deep feature reconstruction loss.
• Show that stereo and temporal constraints improve depth accuracy and VO performance on KITTI.

Proposed method

• Train depth (CNN_D) and visual odometry (CNN_VO) networks jointly from stereo video sequences.
• Use differentiable geometry to synthesize target views via epipolar geometry and bilinear warping, enabling image reconstruction losses.
• Impose an image reconstruction loss combining left-right and temporal consistency for supervision.
• Introduce a deep feature reconstruction loss to provide robust, context-aware supervision beyond raw pixel intensities.
• Apply an edge-aware depth smoothness loss to regularize depth predictions.
• Optionally fuse features from ImageNet, NYUv2-descriptor, or self-embedded depth features within the feature reconstruction term.

Experimental results

Research questions

• RQ1 Can stereo training remove per-frame scale ambiguity in monocular depth and VO estimation and provide metric scale at test time?
• RQ2 Does incorporating temporal information and deep feature-based reconstruction improve depth and VO accuracy beyond color-based photometric losses?
• RQ3 What is the impact of jointly training depth and pose networks on depth quality and frame-to-frame odometry performance on KITTI?

Key findings

• The stereo-based, joint training framework yields metric-scale depth and competitive monocular VO results without external scale supervision.
• Deep feature reconstruction loss improves depth and VO accuracy over pure photometric (color) warp loss.
• Joint depth and VO training with stereo and temporal constraints outperforms prior monocular VO methods and remains competitive with geometric baselines on KITTI.
• Incorporating learned features (from ImageNet or self-supervised depth features) in the warp loss further boosts performance.
• The approach achieves state-of-the-art results among unsupervised methods on KITTI for both single-view depth estimation and frame-to-frame VO.