[Paper Review] GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose

Zhichao Yin, Jianping Shi|arXiv (Cornell University)|Mar 6, 2018

Advanced Vision and Imaging39 references122 citations

TL;DR

GeoNet jointly learns monocular depth, optical flow, and ego-motion from videos in an unsupervised, geometry-driven framework, using a two-stage cascade and adaptive geometric consistency to handle dynamic regions and occlusions; it achieves state-of-the-art results among unsupervised methods on KITTI, comparable to supervised approaches.

ABSTRACT

We propose GeoNet, a jointly unsupervised learning framework for monocular depth, optical flow and ego-motion estimation from videos. The three components are coupled by the nature of 3D scene geometry, jointly learned by our framework in an end-to-end manner. Specifically, geometric relationships are extracted over the predictions of individual modules and then combined as an image reconstruction loss, reasoning about static and dynamic scene parts separately. Furthermore, we propose an adaptive geometric consistency loss to increase robustness towards outliers and non-Lambertian regions, which resolves occlusions and texture ambiguities effectively. Experimentation on the KITTI driving dataset reveals that our scheme achieves state-of-the-art results in all of the three tasks, performing better than previously unsupervised methods and comparably with supervised ones.

Motivation & Objective

• Motivate learning 3D scene geometry from monocular video without ground-truth labels.
• Decompose scene motion into rigid (depth+ego-motion) and non-rigid (object motion) components.
• Enforce geometric consistency to robustly handle occlusions and non-Lambertian regions.
• Propose an end-to-end two-stage architecture to improve unsupervised depth, flow, and pose estimation.
• Evaluate on KITTI showing competitive performance against supervised methods.

Proposed method

• Two-stage cascaded architecture: first stage rigid structure reconstructor (DepthNet + PoseNet) to estimate depth and camera motion, producing rigid flow; second stage ResFlowNet to learn residual non-rigid flow and refine full flow.
• End-to-end differentiable view synthesis losses that compare synthesized views to target frames (photometric loss combining SSIM and L1).
• Edge-aware depth smoothness loss to preserve details while encouraging smooth depth in textureless regions.
• Adaptive geometric consistency loss that mimics forward-backward consistency, selectively enforcing coherence only in non-occluded regions to mitigate occlusions and non-Lambertian effects.
• Joint loss L = sum over scales and frame pairs of: L_rw + lambda_ds*L_ds + L_fw + lambda_fs*L_fs + lambda_gc*L_gc.

Experimental results

Research questions

• RQ1 Can monocular depth, optical flow, and ego-motion be learned jointly in an unsupervised manner from video?
• RQ2 How can static and dynamic scene components be separated and leveraged to improve robustness to occlusions and non-Lambertian regions?
• RQ3 Does enforcing adaptive geometric consistency improve prediction quality for depth, flow, and pose in unsupervised learning?
• RQ4 What are the trade-offs and performance gains of a two-stage architecture (rigid structure vs. non-rigid motion) on KITTI?

Key findings

• GeoNet achieves state-of-the-art results among unsupervised methods on KITTI for depth, flow, and camera pose estimation.
• The two-stage architecture (DepthNet + PoseNet followed by ResFlowNet) effectively decomposes rigid and non-rigid motion, improving overall predictions.
• Adaptive geometric consistency loss improves robustness to occlusions and texture ambiguities by selectively enforcing prediction coherence in non-occluded regions.
• On KITTI, GeoNet’s unsupervised depth estimation and pose estimation are competitive with supervised approaches, and its flow results surpass prior unsupervised baselines.
• Ablation studies show the importance of geometric consistency and residual flow in handling dynamic objects and challenging regions (occlusions, lighting).