[Paper Review] Structured Depth Prediction in Challenging Monocular Video Sequences

Miaomiao Liu, Mathieu Salzmann+1 more|arXiv (Cornell University)|Nov 19, 2015

Advanced Vision and Imaging38 references3 citations

TL;DR

This paper proposes a structured depth prediction framework for monocular video sequences under challenging conditions—non-translational camera motion and dynamic scenes—by modeling depth estimation as a discrete-continuous optimization problem using particle belief propagation in a graphical model. The method extends to two-frame conditional random fields (CRFs) for short-term temporal consistency and a fully-connected CRF for long-range spatio-temporal coherence, achieving state-of-the-art performance on NYUv2, Make3D, and MSR-V3D datasets with improved depth boundary preservation and realism in dynamic scenes.

ABSTRACT

In this paper, we tackle the problem of estimating the depth of a scene from a monocular video sequence. In particular, we handle challenging scenarios, such as non-translational camera motion and dynamic scenes, where traditional structure from motion and motion stereo methods do not apply. To this end, we first study the problem of depth estimation from a single image. In this context, we exploit the availability of a pool of images for which the depth is known, and formulate monocular depth estimation as a discrete-continuous optimization problem, where the continuous variables encode the depth of the superpixels in the input image, and the discrete ones represent relationships between neighboring superpixels. The solution to this discrete-continuous optimization problem is obtained by performing inference in a graphical model using particle belief propagation. To handle video sequences, we then extend our single image model to a two-frame one that naturally encodes short-range temporal consistency and inherently handles dynamic objects. Based on the prediction of this model, we then introduce a fully-connected pairwise CRF that accounts for longer range spatio-temporal interactions throughout a video. We demonstrate the effectiveness of our model in both the indoor and outdoor scenarios.

Motivation & Objective

• Address the challenge of depth estimation in monocular video sequences with non-translational camera motion and dynamic scenes, where traditional SFM and motion stereo methods fail.
• Overcome limitations of single-image depth estimation by incorporating spatio-temporal consistency across video frames.
• Develop a structured prediction model that jointly reasons about depth, motion, and scene structure (static vs. dynamic) in a unified probabilistic framework.
• Improve depth boundary accuracy and temporal smoothness in complex scenes without relying on external annotations or semantic labels.
• Demonstrate superior performance over single-frame methods and prior video-based depth estimation techniques on benchmark datasets.

Proposed method

• Formulate monocular depth estimation as a discrete-continuous optimization problem, where continuous variables represent superpixel depths and discrete variables encode geometric relationships (e.g., occlusion, co-planarity) between neighboring superpixels.
• Perform inference in the resulting higher-order graphical model using particle belief propagation to handle the mixed discrete-continuous inference problem.
• Extend the single-image model to a two-frame conditional random field (CRF) that jointly estimates depth and motion, enforcing spatial coherence and short-term temporal consistency.
• Incorporate a motion-aware energy term in the two-frame CRF to explicitly model moving objects and maintain depth coherence between them and their surroundings.
• Construct a fully-connected pairwise CRF over all pixels in a video sequence to model long-range spatio-temporal dependencies, using Gaussian edge potentials for efficient inference.
• Leverage depth predictions from the two-frame CRF as input to the fully-connected CRF, enabling global refinement of depth maps across the entire video sequence.

Experimental results

Research questions

• RQ1 Can a discrete-continuous graphical model with particle belief propagation effectively estimate depth from a single monocular image without external annotations?
• RQ2 How can depth estimation be extended to video sequences with non-translational camera motion and dynamic objects while preserving spatial and temporal coherence?
• RQ3 To what extent does modeling both short-range (two-frame) and long-range (fully-connected) spatio-temporal dependencies improve depth estimation accuracy in challenging video sequences?
• RQ4 How does the inclusion of a moving foreground classifier impact the realism and accuracy of depth predictions in dynamic scenes?
• RQ5 Does the proposed video-based approach outperform state-of-the-art single-frame and video-based depth estimation methods on standard benchmarks?

Key findings

• The proposed two-frame CRF with a moving foreground classifier significantly improves realism in depth predictions, as evidenced by qualitative results, despite minimal numerical improvement in error metrics.
• On the MSR-V3D dataset, the final video-based model (Ours-Vid) achieved a relative error (rel) of 0.253, log10 error of 0.112, and RMS error of 1.65, outperforming the single-frame baseline and prior video methods.
• For pure rotational sequences in Building 1, the method reduced RMS error from 14.1 (DepthTransfer) to 6.54, demonstrating superior performance on challenging camera motion.
• The fully-connected CRF effectively enforces both spatial and temporal smoothness while preserving depth discontinuities, as shown in qualitative comparisons where the method avoids over-smoothing common in prior approaches.
• The two-frame CRF with motion-aware modeling yields more accurate depth boundaries and better handles occlusions and moving objects compared to models without such components.
• The ablation study confirms that all components of the video-based framework—including the two-frame CRF and the fully-connected CRF—contribute to improved performance, with the full model consistently outperforming ablated variants across all metrics and datasets.