[Paper Review] Deep Patch Visual Odometry

Zachary Teed, Lahav Lipson+1 more|arXiv (Cornell University)|Aug 8, 2022

Robotics and Sensor-Based Localization40 citations

TL;DR

DPVO introduces a sparse patch-based recurrent network for monocular visual odometry with differentiable bundle adjustment, achieving state-of-the-art accuracy while using substantially less memory and running faster than prior methods.

ABSTRACT

We propose Deep Patch Visual Odometry (DPVO), a new deep learning system for monocular Visual Odometry (VO). DPVO uses a novel recurrent network architecture designed for tracking image patches across time. Recent approaches to VO have significantly improved the state-of-the-art accuracy by using deep networks to predict dense flow between video frames. However, using dense flow incurs a large computational cost, making these previous methods impractical for many use cases. Despite this, it has been assumed that dense flow is important as it provides additional redundancy against incorrect matches. DPVO disproves this assumption, showing that it is possible to get the best accuracy and efficiency by exploiting the advantages of sparse patch-based matching over dense flow. DPVO introduces a novel recurrent update operator for patch based correspondence coupled with differentiable bundle adjustment. On Standard benchmarks, DPVO outperforms all prior work, including the learning-based state-of-the-art VO-system (DROID) using a third of the memory while running 3x faster on average. Code is available at https://github.com/princeton-vl/DPVO

Motivation & Objective

• Motivate robust monocular VO without heavy dense-flow computation.
• Develop a sparse patch-based representation that retains robustness.
• Design a recurrent update operator with differentiable bundle adjustment for end-to-end learning.
• Demonstrate efficiency and accuracy improvements over prior learning-based VO/SLAM methods.

Proposed method

• Represent the scene with a set of image patches and a patch graph linking patches to frames.
• Extract multi-scale patch features using a pair of CNNs for matching and context.
• Use a recurrent update operator with correlation, temporal convolution, and soft-aggregation to refine patch trajectories.
• Propose 2D trajectory revisions and confidence weights via a factor head.
• Optimize camera poses and patch depths with a differentiable bundle adjustment layer.
• Train end-to-end with supervised signals for poses and patch-induced flow on synthetic data.

Experimental results

Research questions

• RQ1 Can sparse patch-based matching match or exceed the accuracy of dense-flow deep VO methods?
• RQ2 Does a recurrent patch-based update with differentiable BA provide robustness comparable to, or better than, dense approaches?
• RQ3 How does DPVO balance accuracy, speed, and memory usage relative to prior methods like DROID-SLAM?
• RQ4 What is the impact of patch-based design choices (patch extraction, number of patches, graph connectivity) on VO performance?

Key findings

• DPVO achieves lower average error than prior VO methods across multiple benchmarks (EuRoC, TUM-RGBD, TartanAir, ICL-NUIM).
• DPVO runs 1.5–8.9x faster and uses 57–29% of the memory of prior dense-flow methods like DROID-SLAM in VO mode.
• A 60 FPS variant uses about 4.9 GB memory, and a 120 FPS variant uses around 2.5 GB on the RTX-3090, with relatively stable frame rates.
• Patch-based correspondence with the update operator and differentiable BA yields strong robustness without dense flow.
• Random patch centroid selection performs well, and ablations show the importance of patch-based features and update components.