[Paper Review] Unsupervised Video Depth Estimation Based on Ego-motion and Disparity Consensus

Lingtao Zhou, Jiaojiao Fang+1 more|arXiv (Cornell University)|Sep 3, 2019

Advanced Vision and Imaging23 references4 citations

TL;DR

This paper proposes an unsupervised monocular video depth estimation method that improves depth and ego-motion estimation by integrating left-right disparity consistency and view synthesis losses. By leveraging temporal and stereo consistency with edge-aware smoothness regularization, the approach achieves competitive performance using only 60% of the training data required by baseline methods, outperforming some supervised methods on KITTI and Make3D datasets.

ABSTRACT

Unsupervised learning based depth estimation methods have received more and more attention as they do not need vast quantities of densely labeled data for training which are touch to acquire. In this paper, we propose a novel unsupervised monocular video depth estimation method in natural scenes by taking advantage of the state-of-the-art method of Zhou et al. which jointly estimates depth and camera motion. Our method advances beyond the baseline method by three aspects: 1) we add an additional signal as supervision to the baseline method by incorporating left-right binocular images reconstruction loss based on the estimated disparities, thus the left frame can be reconstructed by the temporal frames and right frames of stereo vision; 2) the network is trained by jointly using two kinds of view syntheses loss and left-right disparity consistency regularization to estimate depth and pose simultaneously; 3) we use the edge aware smooth L2 regularization to smooth the depth map while preserving the contour of the target. Extensive experiments on the KITTI autonomous driving dataset and Make3D dataset indicate the superiority of our algorithm in training efficiency. We can achieve competitive results with the baseline by only 3/5 times training data. The experimental results also show that our method even outperforms the classical supervised methods that using either ground truth depth or given pose for training.

Motivation & Objective

• To develop a self-supervised method for monocular video depth estimation without requiring dense depth annotations.
• To improve training efficiency and depth estimation accuracy by incorporating stereo consistency and view synthesis losses.
• To reduce dependency on large-scale labeled datasets by leveraging unlabeled video sequences.
• To preserve depth map boundaries through edge-aware regularization while smoothing depth predictions.
• To achieve performance comparable or superior to supervised methods using only a fraction of the training data.

Proposed method

• The method extends the Zhou et al. unsupervised depth estimation framework by introducing a left-right binocular image reconstruction loss based on estimated disparities.
• It jointly optimizes depth and ego-motion using two view synthesis losses: one from temporal frames and one from stereo disparity consistency.
• A disparity consistency regularization term ensures that the predicted disparities between left and right views are symmetric and consistent.
• Edge-aware smooth L2 regularization is applied to preserve object boundaries while smoothing depth maps.
• The network is trained end-to-end using unlabeled video sequences, with supervision derived from photometric consistency and stereo consistency.
• The approach leverages both monocular video sequences and stereo image pairs to improve depth estimation robustness.

Experimental results

Research questions

• RQ1 Can stereo consistency and view synthesis losses improve unsupervised depth estimation in monocular video sequences?
• RQ2 How does incorporating left-right disparity consistency affect depth map quality and training convergence?
• RQ3 To what extent can training data be reduced while maintaining or improving performance compared to supervised baselines?
• RQ4 Does edge-aware regularization enhance depth map accuracy at object boundaries?
• RQ5 Can unsupervised methods achieve performance comparable to supervised methods without ground truth depth supervision?

Key findings

• The proposed method achieves competitive depth estimation performance using only 60% of the training data required by the baseline unsupervised method.
• On the KITTI dataset, the method outperforms several classical supervised depth estimation methods that use ground truth depth or known pose.
• The model demonstrates improved generalization and faster convergence due to the addition of stereo consistency and view synthesis losses.
• Edge-aware smoothness regularization significantly enhances depth map quality by preserving sharp boundaries while reducing noise in smooth regions.
• The combination of temporal and stereo view synthesis losses leads to more accurate and consistent depth predictions across sequences.
• The method achieves state-of-the-art performance among unsupervised methods on both KITTI and Make3D benchmarks with minimal supervision.