[Paper Review] Camera Height Doesn't Change: Unsupervised Training for Metric Monocular Road-Scene Depth Estimation

Genki Kinoshita, Ko Nishino|arXiv (Cornell University)|Dec 7, 2023

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes StableCamH, a novel unsupervised monocular depth estimation framework that leverages a learned size prior to infer vehicle dimensions from appearance and aggregate these cues into a single invariant measure—camera height—thereby enabling metric depth estimation without any auxiliary sensors or scale supervision. The method achieves state-of-the-art accuracy on KITTI and Cityscapes by optimizing camera height across frames and epochs, ensuring consistent, scale-aware depth predictions.

ABSTRACT

In this paper, we introduce a novel training method for making any monocular depth network learn absolute scale and estimate metric road-scene depth just from regular training data, i.e., driving videos. We refer to this training framework as FUMET. The key idea is to leverage cars found on the road as sources of scale supervision and to incorporate them in network training robustly. FUMET detects and estimates the sizes of cars in a frame and aggregates scale information extracted from them into an estimate of the camera height whose consistency across the entire video sequence is enforced as scale supervision. This realizes robust unsupervised training of any, otherwise scale-oblivious, monocular depth network so that they become not only scale-aware but also metric-accurate without the need for auxiliary sensors and extra supervision. Extensive experiments on the KITTI and the Cityscapes datasets show the effectiveness of FUMET, which achieves state-of-the-art accuracy. We also show that FUMET enables training on mixed datasets of different camera heights, which leads to larger-scale training and better generalization. Metric depth reconstruction is essential in any road-scene visual modeling, and FUMET democratizes its deployment by establishing the means to convert any model into a metric depth estimator.

Motivation & Objective

• Address the scale ambiguity in unsupervised monocular depth estimation by leveraging prior knowledge of object sizes in road scenes.
• Overcome the brittleness of direct object size priors due to inaccurate localization and domain shifts in real-world scenes.
• Develop a self-supervised training framework that makes any monocular depth estimator scale-aware without requiring GPS, IMU, or depth sensors.
• Enable consistent depth estimation across video sequences by enforcing camera height invariance across frames and training epochs.
• Create a generalizable, pre-trained size prior that estimates vehicle dimensions from appearance alone, usable across datasets without manual annotations.

Proposed method

• Formulate monocular depth estimation as camera height optimization, treating camera height as a shared, invariant physical measure across all frames in a video sequence.
• Introduce a differentiable camera height computation module that derives an unscaled camera height estimate from predicted depth maps.
• Use a novel Learned Size Prior network to predict vehicle dimensions (length, width, height) directly from object appearance, trained on web-sourced car images with annotated dimensions.
• Apply a Silhouette Projector to extract object silhouette heights from depth maps and compare them with predictions from the Learned Size Prior to compute a per-frame scale factor.
• Compute a scaled camera height estimate per frame by applying the scale factor to the unscaled camera height, enabling metric supervision.
• Optimize the camera height across all frames and training epochs using a median operation and weighted moving average, enforcing invariance and stabilizing training.

Experimental results

Research questions

• RQ1 Can camera height invariance across a video sequence be used as a self-supervisory signal to resolve scale ambiguity in monocular depth estimation?
• RQ2 How can object size priors be made robust to localization errors and domain shifts in real-world driving scenes?
• RQ3 Can a learning-based size prior trained on web-scraped data accurately predict vehicle dimensions from appearance alone, without manual annotations?
• RQ4 Does optimizing camera height online during training improve depth estimation accuracy compared to fixed pseudo-supervision?
• RQ5 Can the proposed framework be generalized to work with any monocular depth estimation model and across diverse datasets like KITTI and Cityscapes?

Key findings

• StableCamH achieves state-of-the-art absolute-scale depth estimation performance on both KITTI and Cityscapes benchmarks, outperforming existing weakly-supervised methods.
• The model trained with online camera height optimization achieves higher accuracy than fixed pseudo-supervision, demonstrating the effectiveness of end-to-end invariance learning.
• The fine-tuned camera height strategy further improves performance, showing that additional pre-training enhances accuracy.
• The Learned Size Prior generalizes robustly across datasets, achieving lower absolute relative error in dimension estimation than fixed priors and a comparable 3D detection model (MonoCInIS).
• On KITTI, the Learned Size Prior achieves lower error than the fixed prior (1.59m) and MonoCInIS, especially on unseen domains.
• The method enables fully unsupervised training with no depth or scale supervision, making it applicable to arbitrary in-the-wild driving videos.