[Paper Review] Online Mutual Adaptation of Deep Depth Prediction and Visual SLAM

Shing Yan Loo, Moein Shakeri+3 more|arXiv (Cornell University)|Nov 7, 2021

Robotics and Sensor-Based Localization29 references4 citations

TL;DR

This paper proposes an online mutual adaptation framework that jointly improves monocular depth prediction and visual SLAM by fine-tuning a pre-trained CNN using SLAM-generated keyframes and leveraging the adapted depth to enhance map quality through noise culling and global photometric bundle adjustment. The method reduces depth error and improves SLAM accuracy, especially in unseen environments, by applying Elastic Weight Consolidation (EWC) regularization to prevent catastrophic forgetting during online adaptation.

ABSTRACT

The ability of accurate depth prediction by a convolutional neural network (CNN) is a major challenge for its wide use in practical visual simultaneous localization and mapping (SLAM) applications, such as enhanced camera tracking and dense mapping. This paper is set out to answer the following question: Can we tune a depth prediction CNN with the help of a visual SLAM algorithm even if the CNN is not trained for the current operating environment in order to benefit the SLAM performance? To this end, we propose a novel online adaptation framework consisting of two complementary processes: a SLAM algorithm that is used to generate keyframes to fine-tune the depth prediction and another algorithm that uses the online adapted depth to improve map quality. Once the potential noisy map points are removed, we perform global photometric bundle adjustment (BA) to improve the overall SLAM performance. Experimental results on both benchmark datasets and a real robot in our own experimental environments show that our proposed method improves the overall SLAM accuracy. While regularization has been shown to be effective in multi-task classification problems, we present experimental results and an ablation study to show the effectiveness of regularization in preventing catastrophic forgetting in the online adaptation of depth prediction, a single-task regression problem. In addition, we compare our online adaptation framework against the state-of-the-art pre-trained depth prediction CNNs to show that our online adapted depth prediction CNN outperforms the depth prediction CNNs that have been trained on a large collection of datasets.

Motivation & Objective

• To address the challenge of deploying pre-trained depth prediction CNNs in real-world visual SLAM systems where domain shift degrades performance.
• To enable online adaptation of a depth CNN using SLAM feedback without requiring retraining on large-scale datasets.
• To improve SLAM accuracy by integrating online-adapted depth predictions into global photometric bundle adjustment after culling noisy map points.
• To investigate the effectiveness of regularization techniques—particularly EWC—in preventing catastrophic forgetting during online fine-tuning of a single-task depth regression model.
• To demonstrate that online-adapted depth prediction can surpass state-of-the-art pre-trained models in terms of depth accuracy and SLAM performance on long sequences.

Proposed method

• The framework alternates between two phases: (1) using SLAM to extract keyframes and assess depth prediction quality, and (2) fine-tuning the depth CNN on these keyframes when prediction errors exceed a threshold.
• Noisy 3D map points are removed using the online-adapted depth predictions before performing global photometric bundle adjustment to improve camera tracking and map reconstruction.
• Elastic Weight Consolidation (EWC) is applied to regularize the CNN during online fine-tuning, preserving previously learned knowledge by measuring parameter importance via the Fisher information matrix.
• The method uses a hybrid loss combining the photometric reconstruction loss on new data and an EWC regularization term that penalizes large updates to important parameters.
• The framework is evaluated using both benchmark datasets and real-world robot deployments, comparing performance against pre-trained depth models and ablation variants without regularization.
• Alternative regularization methods—SI and MAS—are also evaluated to compare their effectiveness in preventing forgetting during online adaptation.

Experimental results

Research questions

• RQ1 Can a pre-trained depth CNN be effectively fine-tuned in real time using feedback from a visual SLAM system to improve depth prediction in unseen environments?
• RQ2 Does online adaptation of a depth CNN lead to measurable improvements in visual SLAM accuracy, particularly in long sequences?
• RQ3 Can regularization techniques like EWC prevent catastrophic forgetting during online fine-tuning of a single-task regression model such as depth prediction?
• RQ4 How does the integration of online-adapted depth into global bundle adjustment affect map quality and camera tracking stability?
• RQ5 Does online-adapted depth prediction outperform state-of-the-art pre-trained depth models in terms of depth error and SLAM performance?

Key findings

• The proposed online adaptation framework significantly improves SLAM accuracy in real-world environments by leveraging SLAM feedback to refine depth predictions.
• The use of online-adapted depth for culling noisy map points before global photometric bundle adjustment leads to more accurate camera tracking and improved 3D map reconstruction.
• EWC regularization effectively prevents catastrophic forgetting during online fine-tuning of the depth CNN, outperforming experience replay alone in retaining prior knowledge.
• The online-adapted depth prediction model achieves lower depth error than state-of-the-art pre-trained models, especially after prolonged adaptation on long sequences.
• Ablation studies confirm that regularization is essential for stable online adaptation, with SI and MAS showing comparable or slightly inferior performance to EWC in this context.
• The framework demonstrates robustness and adaptability in diverse real-world settings, showing consistent improvements across multiple evaluation metrics.