[Paper Review] Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video

Jia-Wang Bian, Zhichao Li+5 more|arXiv (Cornell University)|Aug 28, 2019

Advanced Vision and ImagingComputer Science295 citations

TL;DR

The paper proposes a geometry consistency loss and a self-discovered mask to train depth and ego-motion networks from monocular video, achieving state-of-the-art monocular-depth results on KITTI and globally scale-consistent visual odometry without stereo supervision.

ABSTRACT

Recent work has shown that CNN-based depth and ego-motion estimators can be learned using unlabelled monocular videos. However, the performance is limited by unidentified moving objects that violate the underlying static scene assumption in geometric image reconstruction. More significantly, due to lack of proper constraints, networks output scale-inconsistent results over different samples, i.e., the ego-motion network cannot provide full camera trajectories over a long video sequence because of the per-frame scale ambiguity. This paper tackles these challenges by proposing a geometry consistency loss for scale-consistent predictions and an induced self-discovered mask for handling moving objects and occlusions. Since we do not leverage multi-task learning like recent works, our framework is much simpler and more efficient. Comprehensive evaluation results demonstrate that our depth estimator achieves the state-of-the-art performance on the KITTI dataset. Moreover, we show that our ego-motion network is able to predict a globally scale-consistent camera trajectory for long video sequences, and the resulting visual odometry accuracy is competitive with the recent model that is trained using stereo videos. To the best of our knowledge, this is the first work to show that deep networks trained using unlabelled monocular videos can predict globally scale-consistent camera trajectories over a long video sequence.

Motivation & Objective

• Motivate unsupervised depth and ego-motion learning from monocular video without ground-truth depth or poses.
• Address scale-inconsistency across samples and moving objects in static-scene assumptions.
• Introduce geometry-based consistency constraints and a self-discovered mask to improve robustness and efficiency.

Proposed method

• Predict depth maps and relative camera pose from consecutive frames using CNNs.
• Warp and reconstruct one frame from another using predicted depth and pose with differentiable interpolation.
• Introduce a geometry consistency loss L_GC that minimizes depth inconsistency between consecutive frames to enforce scale-consistency.
• Derive a self-discovered mask M from depth inconsistency D_diff to down-weight pixels from dynamic regions and occlusions in the photometric loss.
• Train with a weighted sum of photometric loss, SSIM, edge-aware smoothness, and the geometry consistency loss.
• Evaluate on KITTI with single-scale supervision and demonstrate state-of-the-art depth and competitive long-sequence VO without multi-task supervision.

Experimental results

Research questions

• RQ1 Can depth and ego-motion networks trained on monocular video achieve globally scale-consistent predictions over long sequences?
• RQ2 How can geometry constraints be used to enforce scale-consistency without additional supervision or multi-tasking?
• RQ3 Can a self-discovered mask from reconstruction inconsistency effectively handle dynamic objects and occlusions without extra networks?
• RQ4 How does single-scale supervision compare to multi-scale supervision in unsupervised monocular learning for depth?
• RQ5 What is the comparative performance of monocularly trained VO versus stereo-trained baselines on KITTI?

Key findings

• Depth estimator achieves state-of-the-art performance on KITTI among methods trained on monocular videos.
• Ego-motion predictor yields globally scale-consistent camera trajectories over long sequences, competitive with stereo-trained models.
• Geometry consistency loss reduces scale-inconsistency and propagates frame-to-frame consistency across sequences.
• Self-discovered mask derived from depth inconsistency effectively down-weights dynamic/occluded regions without extra networks.
• Single-scale supervision outperforms multi-scale supervision in their setup, improving robustness and efficiency.
• The approach demonstrates strong depth accuracy and competitive visual odometry, with substantial training efficiency improvements.