[Paper Review] Unsupervised Learning of Geometry with Edge-aware Depth-Normal Consistency

Zhenheng Yang, Peng Wang+3 more|arXiv (Cornell University)|Nov 10, 2017

Advanced Vision and ImagingComputer Science30 references104 citations

TL;DR

This paper presents an unsupervised framework that jointly estimates depth and surface normals from monocular video by enforcing geometry consistency and edge-aware smoothness, outperforming state-of-the-art on KITTI 2015.

ABSTRACT

Learning to reconstruct depths in a single image by watching unlabeled videos via deep convolutional network (DCN) is attracting significant attention in recent years. In this paper, we introduce a surface normal representation for unsupervised depth estimation framework. Our estimated depths are constrained to be compatible with predicted normals, yielding more robust geometry results. Specifically, we formulate an edge-aware depth-normal consistency term, and solve it by constructing a depth-to-normal layer and a normal-to-depth layer inside of the DCN. The depth-to-normal layer takes estimated depths as input, and computes normal directions using cross production based on neighboring pixels. Then given the estimated normals, the normal-to-depth layer outputs a regularized depth map through local planar smoothness. Both layers are computed with awareness of edges inside the image to help address the issue of depth/normal discontinuity and preserve sharp edges. Finally, to train the network, we apply the photometric error and gradient smoothness for both depth and normal predictions. We conducted experiments on both outdoor (KITTI) and indoor (NYUv2) datasets, and show that our algorithm vastly outperforms state of the art, which demonstrates the benefits from our approach.

Motivation & Objective

• Motivate unsupervised learning of scene geometry (depth and normals) from monocular videos.
• Leverage view synthesis as supervision to enforce geometric consistency.
• Incorporate depth-normal consistency as a regularizer to improve depth and normal estimation.
• Address depth discontinuities and low-texture regions via edge-aware smoothness and image gradient terms.

Proposed method

• End-to-end CNN that learns camera motion, depth, and surface normals from monocular video sequences.
• Photometric warping loss based on 3D inverse warping to synthesize target views from source views.
• Edge-aware smoothness loss that respects image gradients to preserve depth discontinuities.
• Image gradient matching loss to encourage sharp depth and better alignment of image gradients.
• Explicit depth2normal and normal2depth layers to enforce geometry consistency between depth and normals.

Experimental results

Research questions

• RQ1 Can depth and surface normals be jointly estimated in an unsupervised manner from monocular video using geometric and photometric constraints?
• RQ2 How does explicit depth-normal geometry regularization affect depth and normal estimation quality?
• RQ3 What is the impact of edge-aware terms on depth smoothness and discontinuities in low-texture regions?

Key findings

• The framework achieves state-of-the-art performance on depth and normal evaluation metrics on KITTI 2015.
• Incorporating depth-normal consistency via dedicated layers improves the quality of both depth and normal maps.
• Edge-aware smoothness and gradient-based losses help preserve depth discontinuities aligned with image edges.
• View synthesis supervision (photometric warping) provides strong geometric signals for learning from monocular video.