[Paper Review] Unsupervised Learning of Depth and Ego-Motion from Video

Tinghui Zhou, Matthew Brown+2 more|arXiv (Cornell University)|Apr 25, 2017

Advanced Vision and Imaging47 references222 citations

TL;DR

An end-to-end unsupervised framework that learns monocular depth and camera ego-motion from unlabeled video by training with a differentiable view-synthesis loss, jointly predicting per-pixel depth and 6-DoF pose with an explainability mask to handle non-idealities.

ABSTRACT

We present an unsupervised learning framework for the task of monocular depth and camera motion estimation from unstructured video sequences. We achieve this by simultaneously training depth and camera pose estimation networks using the task of view synthesis as the supervisory signal. The networks are thus coupled via the view synthesis objective during training, but can be applied independently at test time. Empirical evaluation on the KITTI dataset demonstrates the effectiveness of our approach: 1) monocular depth performing comparably with supervised methods that use either ground-truth pose or depth for training, and 2) pose estimation performing favorably with established SLAM systems under comparable input settings.

Motivation & Objective

• Motivate learning scene geometry and camera motion from unlabeled video sequences without ground-truth depth or pose.
• Develop an end-to-end CNN framework that maps pixels to depth and 6-DoF pose.
• Leverage differentiable image-based rendering to supervise depth and pose via view synthesis.
• Introduce an explainability mask to handle occlusions, non-rigid motion, and other unmodelled factors.
• Demonstrate effectiveness on KITTI in both depth estimation and ego-motion relative to supervised baselines.

Proposed method

• Use a single-view depth network to predict a per-pixel depth map from the target view.
• Use a pose network that takes the target and nearby source views to predict relative camera poses to each source view.
• Compute a differentiable view synthesis loss by warping source views to the target frame using the predicted depth and poses.
• Incorporate a differentiable image-based renderer with bilinear sampling to reconstruct the target view, enabling backpropagation of pose and depth errors.
• Introduce an explainability mask that down-weights regions where view synthesis is unreliable due to dynamics, occlusions, or non-Lambertian effects, with a regularization term to avoid trivial masking.
• Adopt multi-scale training with depth smoothness priors to address gradient locality and encourage plausible depth maps.

Experimental results

Research questions

• RQ1 Can monocular video provide enough supervision to jointly learn depth and ego-motion without ground-truth labels?
• RQ2 How can view synthesis be used as a supervisory signal to train depth and pose networks end-to-end?
• RQ3 What mechanisms (e.g., explainability masks) are necessary to handle occlusions, dynamics, and non-idealities during unsupervised training?
• RQ4 How does the unsupervised approach compare to supervised depth/pose methods on standard benchmarks like KITTI?

Key findings

• The framework learns single-view depth that is competitive with supervised methods that use ground-truth pose or depth for training on KITTI.
• Pose estimation from monocular sequences performs favorably compared with established SLAM systems under comparable input settings.
• The method operates fully unsupervised, requiring only monocular video sequences for training.
• An explainability mask helps mitigate issues from scene dynamics, occlusions, and non-Lambertian surfaces, improving robustness of view synthesis supervision (though ablations show modest gains on KITTI).
• Test-time deployment allows independent use of the depth and pose networks despite joint training.
• Results include qualitative and quantitative comparisons on KITTI; the approach approaches supervised baselines in depth and approaches SLAM-like performance in ego-motion under constrained inputs.