[Paper Review] VA-DepthNet: A Variational Approach to Single Image Depth Prediction

Ce Liu, Suryansh Kumar+3 more|arXiv (Cornell University)|Feb 13, 2023

Advanced Vision and Imaging17 citations

TL;DR

VA-DepthNet introduces a first-order variational constraint in single-image depth prediction, predicting depth gradients and solving a weighted least-squares problem to recover depth, achieving state-of-the-art results on KITTI and NYU while preserving high-frequency details.

ABSTRACT

We introduce VA-DepthNet, a simple, effective, and accurate deep neural network approach for the single-image depth prediction (SIDP) problem. The proposed approach advocates using classical first-order variational constraints for this problem. While state-of-the-art deep neural network methods for SIDP learn the scene depth from images in a supervised setting, they often overlook the invaluable invariances and priors in the rigid scene space, such as the regularity of the scene. The paper's main contribution is to reveal the benefit of classical and well-founded variational constraints in the neural network design for the SIDP task. It is shown that imposing first-order variational constraints in the scene space together with popular encoder-decoder-based network architecture design provides excellent results for the supervised SIDP task. The imposed first-order variational constraint makes the network aware of the depth gradient in the scene space, i.e., regularity. The paper demonstrates the usefulness of the proposed approach via extensive evaluation and ablation analysis over several benchmark datasets, such as KITTI, NYU Depth V2, and SUN RGB-D. The VA-DepthNet at test time shows considerable improvements in depth prediction accuracy compared to the prior art and is accurate also at high-frequency regions in the scene space. At the time of writing this paper, our method -- labeled as VA-DepthNet, when tested on the KITTI depth-prediction evaluation set benchmarks, shows state-of-the-art results, and is the top-performing published approach.

Motivation & Objective

• Motivate SIDP as an ill-posed problem where scene priors and regularity can improve accuracy.
• Propose a variational constraint that enforces depth-gradient regularity while allowing discontinuities.
• Develop a network that predicts depth gradients and confidence weights and recovers depth via a closed-form solution.
• Integrate the variational layer with an encoder–decoder backbone and a multi-stage refinement pipeline to predict metric depth.

Proposed method

• Predict depth-gradient components (Gamma_x, Gamma_y) and confidenceweights (Sigma_x, Sigma_y) from a V-layer that fuses stride-16/32 features.
• Form an over-determined system using first-order differences and a learnable confidence-weighted matrix to solve for unscaled depth Z_u with Z_u* = (P^T Σ^2 P)^{-1} P^T Σ^2 Γ.
• Upsample and refine the V-layer depth maps through a hierarchical three-stage refinement at 1/16, 1/8, and 1/4 resolutions.
• Estimate global scale and shift via a metric layer that regresses two scalars from a pooled feature map to recover metric depth.
• Train with a combination of a scale-invariant depth loss and a variational loss that enforces agreement with depth-gradients.
• Demonstrate improved high-frequency detail preservation and cross-dataset generalization on KITTI, NYU Depth V2, and SUN RGB-D.

Experimental results

Research questions

• RQ1 Does enforcing a first-order variational constraint improve SIDP accuracy beyond purely data-driven approaches?
• RQ2 How do predicted depth gradients and confidence weights affect depth recovery and generalization across datasets?
• RQ3 Can a variational layer integrated with a transformer-based encoder achieve state-of-the-art results on standard SIDP benchmarks?
• RQ4 What is the impact of the V-layer, different backbones, and ablations on performance and efficiency?

Key findings

• On NYU Depth V2, achieves SILog 8.198 and delta1 0.937, outperforming prior art.
• On KITTI Eigen, achieves SILog 6.817 and delta1 0.977, surpassing several state-of-the-art methods.
• On SUN RGB-D, achieves SILog 12.596 and delta1 0.929, demonstrating cross-dataset generalization when trained on NYU Depth V2.
• VA-DepthNet with Swin-L backbone and V-layer delivers strong accuracy with favorable inference time and parameter counts compared to AdaBins and NeWCRFs.
• Ablation studies confirm the efficacy of the V-layer and the confidence-weighted variational formulation over alternatives like plain convolution or self-attention layers.
• The method maintains high-frequency depth details while leveraging scene regularity for better overall depth maps.