[Paper Review] Monocular Depth Estimation with Hierarchical Fusion of Dilated CNNs and Soft-Weighted-Sum Inference

Bo Li, Yuchao Dai+1 more|arXiv (Cornell University)|Aug 2, 2017

Advanced Vision and Imaging24 references8 citations

TL;DR

This paper proposes an end-to-end monocular depth estimation framework that reformulates depth prediction as a multi-category dense classification task using dilated convolutions and hierarchical feature fusion to capture multi-scale depth cues. By employing soft-weighted-sum inference instead of hard-max, the method reduces quantization error and learns a continuous depth probability distribution, achieving state-of-the-art performance on NYU Depth V2 and KITTI datasets with 82.0% accuracy and 0.139 relative error.

ABSTRACT

Monocular depth estimation is a challenging task in complex compositions depicting multiple objects of diverse scales. Albeit the recent great progress thanks to the deep convolutional neural networks (CNNs), the state-of-the-art monocular depth estimation methods still fall short to handle such real-world challenging scenarios. In this paper, we propose a deep end-to-end learning framework to tackle these challenges, which learns the direct mapping from a color image to the corresponding depth map. First, we represent monocular depth estimation as a multi-category dense labeling task by contrast to the regression based formulation. In this way, we could build upon the recent progress in dense labeling such as semantic segmentation. Second, we fuse different side-outputs from our front-end dilated convolutional neural network in a hierarchical way to exploit the multi-scale depth cues for depth estimation, which is critical to achieve scale-aware depth estimation. Third, we propose to utilize soft-weighted-sum inference instead of the hard-max inference, transforming the discretized depth score to continuous depth value. Thus, we reduce the influence of quantization error and improve the robustness of our method. Extensive experiments on the NYU Depth V2 and KITTI datasets show the superiority of our method compared with current state-of-the-art methods. Furthermore, experiments on the NYU V2 dataset reveal that our model is able to learn the probability distribution of depth.

Motivation & Objective

• To address the challenges of multi-scale depth estimation in complex scenes with diverse object scales.
• To overcome limitations of regression-based methods, including data imbalance and sensitivity to depth quantization.
• To improve robustness and accuracy by modeling depth as a classification task with learnable probability distributions.
• To reduce quantization error and enhance generalization through soft-weighted-sum inference.

Proposed method

• Reformulates monocular depth estimation as a multi-category dense labeling problem instead of regression, enabling use of segmentation-style deep learning techniques.
• Employs dilated convolutions in a residual network backbone to expand receptive fields and capture multi-scale context.
• Uses hierarchical feature fusion to combine multi-scale features from different stages of the network for scale-aware depth representation.
• Applies soft-weighted-sum inference to convert discrete depth category scores into continuous depth predictions, minimizing quantization error.
• Trains the network using multinomial logistic loss, which allows the model to implicitly learn the similarity between adjacent depth categories.
• Leverages the probability distribution output to guide depth prediction, improving robustness and continuity.

Experimental results

Research questions

• RQ1 Can monocular depth estimation be effectively reformulated as a multi-category classification task to improve multi-scale depth prediction?
• RQ2 How does hierarchical fusion of dilated features enhance scale-awareness in depth estimation?
• RQ3 Does soft-weighted-sum inference reduce quantization error and improve prediction continuity compared to hard-max inference?
• RQ4 Can a classification-based network learn meaningful depth probability distributions that reflect spatial and depth relationships?

Key findings

• The proposed method achieves 82.0% pixel accuracy and 0.139 relative error (Rel) on the NYU Depth V2 dataset, outperforming state-of-the-art methods.
• Soft-weighted-sum inference reduces error to 0.139 Rel and improves accuracy to 82.0%, compared to 0.142 Rel and 81.82% accuracy with hard-max inference.
• Component ablation shows that both dilated convolutions and skip connections are critical, with removal reducing accuracy by over 3%.
• The network learns a symmetric, Gaussian-like probability distribution over depth categories, indicating it captures depth similarity and proximity.
• Increasing the number of depth bins from 50 to 1000 reduces pixel accuracy from 67% to 7%, but Rel remains stable (~0.14), showing limited depth perception at fine granularity.
• The confusion matrix reveals diagonal dominance and symmetry, confirming that errors primarily occur between neighboring depth classes, validating the need for soft inference.