[Paper Review] Multimodal Scale Consistency and Awareness for Monocular Self-Supervised Depth Estimation

Hemang Chawla, Arnav Varma+2 more|arXiv (Cornell University)|Mar 3, 2021

Advanced Vision and ImagingComputer Science52 references24 citations

TL;DR

This paper proposes a GPS-to-Scale (g2s) loss for monocular self-supervised depth estimation that enforces scale consistency and awareness by leveraging low-frequency, planar GPS data during training—without requiring GPS at inference. The method improves depth estimation accuracy and sharpness, even with sparse or altitude-lacking GPS, outperforming state-of-the-art methods on KITTI and generalizing well to out-of-distribution datasets like Make3D and Cityscapes.

ABSTRACT

Dense depth estimation is essential to scene-understanding for autonomous driving. However, recent self-supervised approaches on monocular videos suffer from scale-inconsistency across long sequences. Utilizing data from the ubiquitously copresent global positioning systems (GPS), we tackle this challenge by proposing a dynamically-weighted GPS-to-Scale (g2s) loss to complement the appearance-based losses. We emphasize that the GPS is needed only during the multimodal training, and not at inference. The relative distance between frames captured through the GPS provides a scale signal that is independent of the camera setup and scene distribution, resulting in richer learned feature representations. Through extensive evaluation on multiple datasets, we demonstrate scale-consistent and -aware depth estimation during inference, improving the performance even when training with low-frequency GPS data.

Motivation & Objective

• Address scale inconsistency in monocular self-supervised depth estimation, a key limitation in autonomous driving applications.
• Overcome the scale-ambiguity problem inherent in monocular vision by introducing a multimodal learning signal from GPS data.
• Enable metric-scale depth predictions without requiring LiDDR or stereo supervision during inference.
• Demonstrate robustness to low-frequency and 2D GPS (without altitude), increasing practical deployability.
• Improve generalization to out-of-distribution datasets by learning scene- and camera-agnostic depth representations.

Proposed method

• Propose a dynamically-weighted GPS-to-Scale (g2s) loss that compares predicted relative translation from the ego-motion network with GPS-measured relative translation.
• Use the perspective projection model to link depth and ego-motion predictions, ensuring geometric consistency.
• Apply the g2s loss only during training, with GPS data used solely as a scale signal to enforce scale consistency and awareness.
• Implement an exponentially increasing weighting schedule for the g2s loss over training epochs to improve convergence and performance.
• Integrate the g2s loss with standard appearance-based photometric, smoothness, and auto-masking losses in a self-supervised framework.
• Train the depth and ego-motion networks end-to-end using a multimodal loss that combines visual appearance and GPS-derived scale signals.

Experimental results

Research questions

• RQ1 Can GPS-derived relative translation provide a reliable scale signal for monocular self-supervised depth estimation?
• RQ2 How does the performance of the g2s loss vary with GPS frequency and the absence of altitude information?
• RQ3 To what extent does the g2s loss improve scale consistency and awareness across diverse scenes and datasets?
• RQ4 Can the proposed method generalize to out-of-distribution datasets like Make3D and Cityscapes without fine-tuning?
• RQ5 Does the multimodal self-supervised framework lead to richer, more transferable feature representations compared to appearance-only baselines?

Key findings

• The proposed g2s loss significantly improves scale consistency, reducing the standard deviation of depth scale factors to 0.85 on Make3D and 1.22 on Cityscapes, compared to 17.24 and 22.44 for baseline methods.
• The mean depth scale factor is 2.81 on Make3D and 4.01 on Cityscapes, indicating strong scale awareness, with values much closer to 1 than for competing methods.
• Even with low-frequency GPS (as low as 1 Hz) and without altitude, the method achieves state-of-the-art performance on the KITTI Eigen split, with an Abs Rel error of 0.112 and a δ < 1.253 accuracy of 0.981.
• The method outperforms Monodepth2 and other SOTA approaches on the KITTI depth benchmark, achieving a 14.16 SILog error and 3.65 SqRel error with the M+G training setup.
• Qualitative results show finer object delineation—such as building structures, tree trunks, and vehicle details—on both Make3D and Cityscapes, confirming improved feature learning.
• The ablation study confirms that the exponentially weighted g2s loss (Eq. 5) yields the best performance, with a 0.112 Abs Rel error and 0.981 δ < 1.253 accuracy, outperforming constant and linear weighting strategies.