[Paper Review] From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation

Jin Han Lee, Myung Kyu Han+2 more|arXiv (Cornell University)|Jul 24, 2019

Advanced Vision and Imaging49 references483 citations

TL;DR

The paper introduces multi-scale local planar guidance (LPG) layers placed at multiple decoding stages to guide dense feature maps to full-resolution depth, achieving state-of-the-art monocular depth estimation on NYU Depth V2 and KITTI. It also provides extensive ablations validating LPG effectiveness.

ABSTRACT

Estimating accurate depth from a single image is challenging because it is an ill-posed problem as infinitely many 3D scenes can be projected to the same 2D scene. However, recent works based on deep convolutional neural networks show great progress with plausible results. The convolutional neural networks are generally composed of two parts: an encoder for dense feature extraction and a decoder for predicting the desired depth. In the encoder-decoder schemes, repeated strided convolution and spatial pooling layers lower the spatial resolution of transitional outputs, and several techniques such as skip connections or multi-layer deconvolutional networks are adopted to recover the original resolution for effective dense prediction. In this paper, for more effective guidance of densely encoded features to the desired depth prediction, we propose a network architecture that utilizes novel local planar guidance layers located at multiple stages in the decoding phase. We show that the proposed method outperforms the state-of-the-art works with significant margin evaluating on challenging benchmarks. We also provide results from an ablation study to validate the effectiveness of the proposed method.

Motivation & Objective

• Motivate accurate monocular depth estimation as an ill-posed problem and leverage global and local cues.
• Introduce LPG layers at multiple decoding stages to provide explicit geometric guidance.
• Show that LPG-based guidance improves depth accuracy across indoor and outdoor benchmarks.
• Demonstrate robustness and generalization by evaluating with multiple backbone encoders (ResNet, DenseNet, etc.).
• Provide ablation analysis to validate the contribution of LPG and the training loss design.

Proposed method

• Use an encoder–decoder backbone with a dense feature extractor (backbone network).
• Insert a contextual extractor (ASPP with dilation rates) after the backbone.
• Embed local planar guidance (LPG) layers at decoding stages with resolutions H/8, H/4, H/2 to generate 4D plane coefficients per k×k patch.
• Convert plane coefficients to local depth cues via ray–plane intersection to obtain per-patch depth guidance.
• Concatenate LPG outputs from multiple scales with 1×1 reduced features and predict final depth through a final convolutional layer.
• Train with a scale-aware loss based on log-depth error (SiLog variant) and a final loss L = alpha * sqrt(D(g)).
• Experiment with multiple backbones (ResNet-101/ResNext-101/DenseNet-161) and datasets (NYU Depth V2, KITTI) to demonstrate effectiveness.

Experimental results

Research questions

• RQ1 Does incorporating multi-scale LPG layers improve monocular depth estimation over strong baselines?
• RQ2 How do LPG layers at different decoding stages contribute to final depth quality?
• RQ3 What is the impact of different backbone encoders on the performance of LPG-enabled networks?
• RQ4 How does the proposed loss formulation influence training convergence and accuracy?
• RQ5 Are the gains consistent across indoor (NYU) and outdoor (KITTI) datasets?

Key findings

• The LPG-equipped network achieves state-of-the-art results on NYU Depth V2 and KITTI benchmarks, outperforming previous methods on most metrics.
• Ablation shows LPG layers provide the most significant performance gains among core components.
• DenseNet-161 as backbone yields top results on NYU, while ResNext-101 can be strongest on KITTI, illustrating dataset-dependent advantages.
• The method yields notable improvements in inlier metrics (e.g., delta thresholds) and depth accuracy metrics across evaluated splits.
• Even with lightweight backbones (e.g., MobileNetV2), the approach maintains competitive performance with substantial parameter-efficient gains.
• The qualitative results exhibit clearer object boundaries and better local detail preservation compared with competitive methods.