[Paper Review] Deep Virtual Stereo Odometry: Leveraging Deep Depth Prediction for Monocular Direct Sparse Odometry

Nan Yang, Rui Wang+2 more|arXiv (Cornell University)|Jul 6, 2018

Advanced Vision and Imaging36 references8 citations

TL;DR

This paper proposes Deep Virtual Stereo Odometry (DVSO), a monocular visual odometry system that leverages deep monocular depth prediction to overcome scale drift and motion parallax limitations in geometric VO. By using a semi-supervised deep network to refine disparity predictions and integrating them as virtual stereo measurements in Direct Sparse Odometry, DVSO achieves metric accuracy comparable to stereo methods using only a single camera.

ABSTRACT

Monocular visual odometry approaches that purely rely on geometric cues are prone to scale drift and require sufficient motion parallax in successive frames for motion estimation and 3D reconstruction. In this paper, we propose to leverage deep monocular depth prediction to overcome limitations of geometry-based monocular visual odometry. To this end, we incorporate deep depth predictions into Direct Sparse Odometry (DSO) as direct virtual stereo measurements. For depth prediction, we design a novel deep network that refines predicted depth from a single image in a two-stage process. We train our network in a semi-supervised way on photoconsistency in stereo images and on consistency with accurate sparse depth reconstructions from Stereo DSO. Our deep predictions excel state-of-the-art approaches for monocular depth on the KITTI benchmark. Moreover, our Deep Virtual Stereo Odometry clearly exceeds previous monocular and deep learning based methods in accuracy. It even achieves comparable performance to the state-of-the-art stereo methods, while only relying on a single camera.

Motivation & Objective

• Address the inherent scale ambiguity and motion parallax dependency in monocular visual odometry.
• Overcome the limitations of purely geometric monocular VO by incorporating deep learning-based depth estimation.
• Develop a semi-supervised training approach for depth prediction that avoids costly LiDAR data collection.
• Integrate deep depth predictions into a direct sparse odometry framework to enable metric-scale trajectory estimation.
• Achieve performance on par with stereo-based visual odometry systems using only a single camera.

Proposed method

• Train a two-stage stacked residual network for monocular depth prediction using self-supervised photometric consistency and sparse depth supervision from Stereo DSO.
• Use predicted depth maps as virtual stereo disparities to create synthetic stereo constraints in the DSO optimization pipeline.
• Integrate virtual stereo terms into windowed direct bundle adjustment to enforce consistency between predicted depth and observed image intensities.
• Leverage photoconsistency loss and left-right consistency in stereo image pairs to supervise depth prediction without full LiDAR supervision.
• Use sparse 3D reconstructions from Stereo DSO as weak supervision to refine depth predictions during training.
• Tune the virtual stereo baseline parameter to further improve performance, enabling outperformance of state-of-the-art stereo methods.

Experimental results

Research questions

• RQ1 Can deep monocular depth prediction effectively mitigate scale drift in monocular visual odometry?
• RQ2 To what extent can semi-supervised depth learning, using only stereo images and sparse DSO reconstructions, achieve performance comparable to supervised methods?
• RQ3 Can virtual stereo constraints derived from deep depth predictions improve the accuracy of direct sparse odometry in a monocular setup?
• RQ4 Does the proposed method achieve performance on par with stereo-based visual odometry systems while relying solely on a single camera?
• RQ5 How does the system generalize to diverse sequences and camera trajectories beyond the training domain?

Key findings

• DVSO achieves translational and rotational errors on the KITTI benchmark that are comparable to state-of-the-art stereo methods like Stereo LSD-SLAM and Stereo DSO.
• With virtual baseline tuning, DVSO outperforms all evaluated monocular and stereo methods, including Stereo LSD-SLAM and ORB-SLAM2 without loop closure.
• The proposed semi-supervised depth network outperforms state-of-the-art monocular depth estimation methods on the KITTI benchmark.
• DVSO significantly outperforms end-to-end deep learning-based VO systems such as DeepVO, UnDeepVO, SfMLearner, and Yin et al. [46] in terms of both translation and rotation error.
• The method generalizes well to unseen sequences, as demonstrated by trajectory estimation on the Cityscapes Frankfurt sequence in the supplementary material.
• The integration of deep depth predictions as virtual stereo constraints enables metric-scale estimation in a monocular system, effectively eliminating scale drift.