[Paper Review] Unsupervised Monocular Depth Learning in Dynamic Scenes

Hanhan Li, Ariel Gordon+3 more|arXiv (Cornell University)|Oct 30, 2020

Advanced Vision and ImagingComputer Science43 references57 citations

TL;DR

The paper learns depth, ego-motion, and a dense 3D object translation field from monocular video without any auxiliary signals, using a novel motion regularization (L_g1 and L_{1/2}) to handle dynamic scenes, achieving state-of-the-art or competitive results across Cityscapes, KITTI, Waymo Open Dataset, and YouTube videos.

ABSTRACT

We present a method for jointly training the estimation of depth, ego-motion, and a dense 3D translation field of objects relative to the scene, with monocular photometric consistency being the sole source of supervision. We show that this apparently heavily underdetermined problem can be regularized by imposing the following prior knowledge about 3D translation fields: they are sparse, since most of the scene is static, and they tend to be constant for rigid moving objects. We show that this regularization alone is sufficient to train monocular depth prediction models that exceed the accuracy achieved in prior work for dynamic scenes, including methods that require semantic input. Code is at https://github.com/google-research/google-research/tree/master/depth_and_motion_learning .

Motivation & Objective

• Motivate and address depth estimation in highly dynamic scenes using only monocular video supervision.
• Jointly learn depth, ego-motion, and a dense 3D translation field for moving objects.
• Regularize residual object motion to enable robust depth prediction without semantics or stereo.

Proposed method

• Encoder-decoder depth network predicts per-frame depth.
• Motion network predicts a dense 3D object translation field T_obj(u,v) and a 6D ego-motion vector M_ego.
• Depth and motion networks consume the two frames: depth independent per frame, motion using both frames with depth as an additional input channel.
• Motion regularization L_reg,mot combines group smoothness L_g1 and L_{1/2} sparsity to enforce piecewise-constant motion on rigid objects.
• Edge-aware depth regularization and consistency losses: L_reg,dep, L_rgb (photometric), and L_cyc (motion cycle consistency).
• A differentiable view transformer uses K, R, and T to warp between frames for self-supervision.

Experimental results

Research questions

• RQ1 Can monocular videos supervise learning of depth, ego-motion, and dense object motion without semantic or stereo cues?
• RQ2 How can residual motion fields be regularized to handle dynamic scenes with moving objects while preserving depth accuracy?
• RQ3 What are the performance gains on standard dynamic-scene benchmarks (Cityscapes, KITTI, Waymo) and in-the-wild videos when using the proposed regularization?
• RQ4 Does including the predicted depth as an extra input channel to the motion network improve motion estimation?
• RQ5 What is the effect of using L_{1/2} versus L_{1} sparsity in the motion regularization?

Key findings

• Achieves state-of-the-art unsupervised depth on Cityscapes and Waymo Open Dataset, and competitive results on KITTI without using semantic inputs.
• Depth is learned per-frame, while motion is learned from frame pairs as a 3D translation field plus ego-motion.
• Two-part motion regularization (L_g1 and self-normalizing L_{1/2}) enforces sparsity and piecewise-constant motion within moving objects, enabling robust depth despite dynamics.
• Inferences run at about 5.3 ms per frame (480x192) on a V100, enabling ~190 FPS (unoptimized).
• Ablation shows that removing depth input to the motion network or using L_{1} instead of L_{1/2} degrades performance on Cityscapes; adding pretrained masks provides modest gains or no extra benefit depending on setting.
• Results by dataset: Cityscapes Ablation shows Abs Rel 0.119, Sq Rel 1.29, RMSE 6.98, RMSE log 0.190; KITTI (Eigen split) Abs Rel 0.130, Sq Rel 0.950, RMSE 5.138, RMSE log 0.209, δ<1.25 0.843, δ<1.25^2 0.948, δ<1.25^3 0.978; Waymo Open Dataset (no mask) Abs Rel 0.162, Sq Rel 1.711, RMSE 7.833, RMSE log 0.223 (with mask: Abs Rel 0.157, Sq Rel 1.531, RMSE 7.090, RMSE log 0.205).