[Paper Review] D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry

Nan Yang, Lukas von Stumberg+2 more|arXiv (Cornell University)|Mar 2, 2020

Advanced Vision and Imaging85 references33 citations

TL;DR

D3VO integrates deep depth, pose, and uncertainty predictions into a monocular direct VO pipeline, achieving state-of-the-art results on KITTI and EuRoC without external depth supervision.

ABSTRACT

We propose D3VO as a novel framework for monocular visual odometry that exploits deep networks on three levels -- deep depth, pose and uncertainty estimation. We first propose a novel self-supervised monocular depth estimation network trained on stereo videos without any external supervision. In particular, it aligns the training image pairs into similar lighting condition with predictive brightness transformation parameters. Besides, we model the photometric uncertainties of pixels on the input images, which improves the depth estimation accuracy and provides a learned weighting function for the photometric residuals in direct (feature-less) visual odometry. Evaluation results show that the proposed network outperforms state-of-the-art self-supervised depth estimation networks. D3VO tightly incorporates the predicted depth, pose and uncertainty into a direct visual odometry method to boost both the front-end tracking as well as the back-end non-linear optimization. We evaluate D3VO in terms of monocular visual odometry on both the KITTI odometry benchmark and the EuRoC MAV dataset.The results show that D3VO outperforms state-of-the-art traditional monocular VO methods by a large margin. It also achieves comparable results to state-of-the-art stereo/LiDAR odometry on KITTI and to the state-of-the-art visual-inertial odometry on EuRoC MAV, while using only a single camera.

Motivation & Objective

• Motivate and enable robust monocular VO by leveraging deep predictions for depth, pose, and uncertainty.
• Develop a self-supervised depth/pPose network trained on stereo videos with brightness alignment and photometric uncertainty.
• Integrate deep depth, pose, and uncertainty into a direct VO framework to improve front-end tracking and back-end optimization.
• Evaluate D3VO on KITTI Odometry and EuRoC MAV to demonstrate competitive performance with stereo/LiDAR and VIO methods.

Proposed method

• Propose self-supervised DepthNet and PoseNet predicting depth D, relative pose T, and photometric uncertainty Sigma.
• Introduce affine brightness transformation parameters to align illumination between training frames.
• Model per-pixel photometric uncertainty as aleatoric uncertainty to weight residuals in training and VO optimization.
• Incorporate a virtual stereo term and a pose energy term into a sparse photometric bundle adjustment framework.
• Use the network predictions as metric scale initialization, pose priors, and weighting in the energy function to guide tracking and optimization.

Experimental results

Research questions

• RQ1 Can a self-supervised monocular network predict depth with metric scale from stereo supervision and improve VO when integrated into a direct pipeline?
• RQ2 Does incorporating predicted depth, pose, and photometric uncertainty improve front-end tracking and back-end optimization for monocular VO?
• RQ3 How does depth/uncertainty/pPose integration compare to state-of-the-art monocular, stereo, and VIO methods on KITTI and EuRoC?

Key findings

• The self-supervised network with brightness alignment and photometric uncertainty outperforms Monodepth2 on KITTI Eigen depth evaluation.
• D3VO achieves state-of-the-art monocular VO results on KITTI Odometry test sequences, outperforming many monocular baselines and approaching stereo/LiDAR performance.
• On EuRoC MAV, D3VO delivers competitive monocular VO results and robustness, closely matching end-to-end and hybrid approaches.
• Incorporating deep depth, deep pose, and deep uncertainty yields substantial gains over ablated variants (Dd, Dp, Du).
• The method attains comparable performance to state-of-the-art VIO methods on EuRoC while using only a single camera.