[Paper Review] DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras

Zachary Teed, Jia Deng|arXiv (Cornell University)|Aug 24, 2021

Robotics and Sensor-Based LocalizationEngineering53 references233 citations

TL;DR

DROID-SLAM introduces a differentiable recurrent optimization SLAM system that jointly updates camera poses and per-pixel depth using a Dense Bundle Adjustment layer, achieving state-of-the-art accuracy and robustness across monocular, stereo, and RGB-D inputs without retraining.

ABSTRACT

We introduce DROID-SLAM, a new deep learning based SLAM system. DROID-SLAM consists of recurrent iterative updates of camera pose and pixelwise depth through a Dense Bundle Adjustment layer. DROID-SLAM is accurate, achieving large improvements over prior work, and robust, suffering from substantially fewer catastrophic failures. Despite training on monocular video, it can leverage stereo or RGB-D video to achieve improved performance at test time. The URL to our open source code is https://github.com/princeton-vl/DROID-SLAM.

Motivation & Objective

• Motivate robust, accurate visual SLAM across monocular, stereo, and RGB-D modalities.
• Develop a deep learning SLAM framework that jointly optimizes pose and dense depth.
• Enable cross-modal generalization so monocular training benefits stereo and RGB-D testing.
• Incorporate a differentiable DBA layer to fuse classical geometry with learned updates.

Proposed method

• Use an end-to-end differentiable pipeline with recurrent update iterations on camera poses and inverse depth.
• Represent the problem with a frame graph capturing covisible frames and perform updates via a 3x3 ConvGRU.
• Compute dense correspondences and correlation features from image pairs using a RAFT-inspired correlation pyramid.
• Predict flow revisions and confidence maps, then apply a differentiable Dense Bundle Adjustment to obtain pose and depth updates.
• Train with pose and flow supervision, fixing gauge freedom by anchoring the first two poses in monocular training.
• Extend to stereo and RGB-D by adjusting the DBA layer and incorporating depth measurements as needed.

Experimental results

Research questions

• RQ1 Can a deep learning SLAM system achieve high accuracy while maintaining robustness across monocular, stereo, and RGB-D inputs?
• RQ2 Does iterative, differentiable optimization (DROID) can jointly refine camera poses and dense depth over multiple frames?
• RQ3 How does incorporating a dense bundle adjustment layer affect accuracy and failure rates compared to prior SLAM approaches?
• RQ4 To what extent can monocularly trained models generalize to stereo and RGB-D testing without retraining?
• RQ5 What are the performance characteristics (speed, memory) of a real-time end-to-end deep SLAM system on standard benchmarks?

Key findings

• Achieves state-of-the-art accuracy across multiple datasets and modalities, with large relative improvements over prior work.
• Demonstrates high robustness with substantially fewer catastrophic failures across datasets like ETH-3D, TartanAir, EuRoC, and TUM-RGBD.
• Exhibits strong generalization: a model trained only on monocular data can effectively leverage stereo or RGB-D input at test time without retraining.
• On TartanAir monocular, the method reduces error by 62% over the best prior result; on stereo, 60% reduction.
• On ETH-3D RGB-D, ranks 1st in AUC and tracks 30/32 RGB-D datasets, outperforming the next best by a notable margin.
• Outperforms classical SLAM baselines and many deep-learning rivals across Euroc, TUM-RGBD, and ETH-3D benchmarks.