[Paper Review] Geo-Supervised Visual Depth Prediction

Xiaohan Fei, Alex Wong+1 more|arXiv (Cornell University)|Jul 30, 2018

Advanced Vision and Imaging45 references4 citations

TL;DR

This paper proposes geo-supervised visual depth prediction by leveraging gravity as a supervisory signal to improve monocular depth estimation. By enforcing that surface normals of geo-referenced objects (e.g., roads, buildings) align with or are orthogonal to gravity, the method enhances depth prediction via a learnable, class-aware regularizer, achieving state-of-the-art performance on KITTI and VISMA benchmarks without requiring semantic segmentation at test time.

ABSTRACT

We propose using global orientation from inertial measurements, and the bias it induces on the shape of objects populating the scene, to inform visual 3D reconstruction. We test the effect of using the resulting prior in depth prediction from a single image, where the normal vectors to surfaces of objects of certain classes tend to align with gravity or be orthogonal to it. Adding such a prior to baseline methods for monocular depth prediction yields improvements beyond the state-of-the-art and illustrates the power of gravity as a supervisory signal.

Motivation & Objective

• To improve monocular depth prediction by incorporating global orientation from inertial measurements as a supervisory signal.
• To develop a method that selectively applies gravity-based priors to geo-referenced object classes (e.g., roads, buildings) to improve depth estimation accuracy.
• To demonstrate that gravity, as a globally consistent reference, can serve as a powerful regularizer in self-supervised depth learning.
• To validate the approach on diverse benchmarks, including KITTI and indoor VISMA datasets, under varying motion conditions.
• To enable deployment on mobile and handheld devices by integrating with visual-inertial odometry (VIO) systems.

Proposed method

• The method trains a deep neural network end-to-end to predict disparity from a single image and an estimate of gravity in the camera frame.
• A class-specific regularizer is applied to regions identified as geo-referenced objects (e.g., roads, buildings) using semantic segmentation masks during training.
• The regularizer enforces that surface normals of these objects align with or are orthogonal to gravity, modeled via a differentiable loss term.
• For stereo training, the method uses calibrated stereo pairs with differentiable loss functions applied selectively to geo-referenced classes.
• For monocular video training, the method minimizes reprojection error while incorporating the gravity-induced normal regularizer.
• The approach is integrated into existing state-of-the-art models (e.g., GeoNet, OursVIO) and extended with VIO for real-time, handheld deployment.

Experimental results

Research questions

• RQ1 Can gravity serve as an effective supervisory signal for monocular depth prediction in the absence of explicit 3D supervision?
• RQ2 How does enforcing normal alignment with gravity improve depth prediction accuracy on standard benchmarks?
• RQ3 Can the method generalize to challenging indoor environments with complex motion and limited parallax?
• RQ4 Does the inclusion of gravity-based regularization improve performance when combined with visual-inertial odometry (VIO)?
• RQ5 Can the method be applied without requiring semantic segmentation at inference time, despite using it during training?

Key findings

• On the KITTI benchmark, the proposed method outperforms state-of-the-art methods, achieving a relative error (AbsRel) of 0.356 and a root mean square error (RMSE) of 4.517 in monocular depth prediction.
• When applied to the VISMA indoor dataset, the OursVIO++ model achieved an AbsRel of 0.105 and RMSE of 0.421, significantly outperforming the baseline GeoNet (AbsRel: 0.157, RMSE: 0.518).
• The method demonstrated consistent improvement across all metrics on both KITTI and VISMA, including δ<1.25, δ<1.25², and δ<1.25³ accuracy.
• Qualitative results show that the gravity prior effectively reduces holes and improves depth estimation on planar surfaces such as chair backs and walls, especially under extreme viewpoints.
• Ablation studies confirm that the gravity-based regularizer contributes significantly to performance gains, even when combined with VIO-based pose estimation.
• The method generalizes well to unseen domains, as demonstrated by directly applying a KITTI-trained model to the Make3D dataset without fine-tuning.