[Paper Review] DF-VO: What Should Be Learnt for Visual Odometry?

Huangying Zhan, Chamara Saroj Weerasekera+3 more|arXiv (Cornell University)|Mar 1, 2021

Advanced Vision and ImagingComputer Science55 references26 citations

TL;DR

DF-VO combines deep depth and optical flow predictions with multi-view geometry to address scale drift and dynamic scenes in monocular visual odometry, achieving state-of-the-art results on KITTI.

ABSTRACT

Multi-view geometry-based methods dominate the last few decades in monocular Visual Odometry for their superior performance, while they have been vulnerable to dynamic and low-texture scenes. More importantly, monocular methods suffer from scale-drift issue, i.e., errors accumulate over time. Recent studies show that deep neural networks can learn scene depths and relative camera in a self-supervised manner without acquiring ground truth labels. More surprisingly, they show that the well-trained networks enable scale-consistent predictions over long videos, while the accuracy is still inferior to traditional methods because of ignoring geometric information. Building on top of recent progress in computer vision, we design a simple yet robust VO system by integrating multi-view geometry and deep learning on Depth and optical Flow, namely DF-VO. In this work, a) we propose a method to carefully sample high-quality correspondences from deep flows and recover accurate camera poses with a geometric module; b) we address the scale-drift issue by aligning geometrically triangulated depths to the scale-consistent deep depths, where the dynamic scenes are taken into account. Comprehensive ablation studies show the effectiveness of the proposed method, and extensive evaluation results show the state-of-the-art performance of our system, e.g., Ours (1.652%) v.s. ORB-SLAM (3.247%}) in terms of translation error in KITTI Odometry benchmark. Source code is publicly available at: \href{https://github.com/Huangying-Zhan/DF-VO}{DF-VO}.

Motivation & Objective

• Motivate a robust monocular VO system that leverages both deep learning and geometry.
• Sample high-quality correspondences from dense optical flow for reliable pose estimation.
• Use scale-consistent depth predictions to recover and maintain global scale over long videos.
• Handle dynamic scenes with an iterative scale recovery and robust correspondence selection.
• Provide a hybrid tracker selection framework between Epipolar Geometry and PnP-based tracking.

Proposed method

• Predict dense optical flow and single-view depth for each image pair.
• Select high-quality 2D-2D correspondences using forward-backward flow consistency.
• Estimate motion with two trackers: Epipolar Geometry-based E-tracker and PnP-tracker.
• Recover scale by aligning triangulated depths to scale-consistent network depths, with an iterative refinement for dynamics.
• Use Geometric Robust Information Criterion (GRIC) to choose between trackers and apply RANSAC for robustness.
• Train depth and pose networks in a self-supervised framework using photometric and depth consistency losses.

Experimental results

Research questions

• RQ1 How can deep predictions of depth and optical flow be integrated with classic multi-view geometry for robust monocular VO?
• RQ2 Can scale drift be mitigated by aligning triangulated depths with scale-consistent deep depth predictions?
• RQ3 What strategies best handle dynamic scenes in monocular VO to maintain accuracy over long trajectories?
• RQ4 Does a hybrid E-tracker and PnP-tracker with GRIC-based model selection improve robustness over using a single geometry model?
• RQ5 How does iterative scale recovery perform in highly dynamic environments?

Key findings

• DF-VO achieves state-of-the-art translation error on KITTI Odometry with 1.652% versus 3.247% for ORB-SLAM.
• A bi-directional flow consistency-based sampling scheme improves 2D-2D correspondences quality.
• Scale-consistent depth predictions enable robust scale recovery and tracking over long sequences.
• An iterative scale recovery method improves performance in dynamic scenes by filtering depth-flow pairs for consistency.
• GRIC-based model selection effectively switches between E-tracker and PnP-tracker to avoid degeneracies.
• Comprehensive ablation studies demonstrate the effectiveness of the proposed components.