[Paper Review] DeepV2D: Video to Depth with Differentiable Structure from Motion

Zachary Teed, Jia Deng|arXiv (Cornell University)|Dec 11, 2018

Advanced Vision and Imaging65 references89 citations

TL;DR

DeepV2D is an end-to-end differentiable pipeline that alternates depth and motion estimation to predict depth from video, integrating classical SfM geometry as trainable modules.

ABSTRACT

We propose DeepV2D, an end-to-end deep learning architecture for predicting depth from video. DeepV2D combines the representation ability of neural networks with the geometric principles governing image formation. We compose a collection of classical geometric algorithms, which are converted into trainable modules and combined into an end-to-end differentiable architecture. DeepV2D interleaves two stages: motion estimation and depth estimation. During inference, motion and depth estimation are alternated and converge to accurate depth. Code is available https://github.com/princeton-vl/DeepV2D.

Motivation & Objective

• Motivate depth estimation from video sequences leveraging geometric structure from motion.
• Propose a differentiable architecture that interleaves motion (Flow-SE3) and depth modules.
• Show that block-coordinate descent between depth and motion converges to accurate depth.
• Demonstrate cross-dataset generalization and competitive performance against state-of-the-art methods.

Proposed method

• Depth Module builds a cost volume across multiple views using learned 2D features and differentiable backprojection, followed by 3D matching and differentiable soft-argmax to produce depth.
• Motion Module predicts pose updates by minimizing a differentiable Gauss-Newton step on per-pixel geometric reprojection errors, using Flow-SE3 to map 2D correspondences to SE(3) updates.
• Forward pass alternates between depth and motion modules, enabling iteration-based refinement similar to block coordinate descent.
• Camera geometry is formalized with projection and backprojection operators, and relative pose G_ij = G_j G_i^{-1} to relate views.
• Supervision combines depth L1 loss with edge-aware smoothness and a photometric-style pose loss via robust Huber terms.

Experimental results

Research questions

• RQ1 Can a differentiable architecture integrate classical SfM steps (feature matching, PnP, MVS) for depth from video?
• RQ2 Does alternating depth and motion updates converge to accurate depth without ground-truth camera poses?
• RQ3 How well does the model generalize across datasets and varying numbers of input frames?
• RQ4 What is the impact of initialization strategy and number of iterations on depth accuracy?
• RQ5 Can the approach compete with state-of-the-art multi-view and single-view depth estimators on benchmarks like NYU, ScanNet, SUN3D, and KITTI?

Key findings

• DeepV2D outperforms strong baselines such as DeepTAM, DeMoN, BA-Net, and MVSNet on multiple benchmarks.
• The method generalizes well across datasets and can surpass methods even when trained on different data (cross-dataset generalization).
• Few iterations are needed for convergence, and more input frames improve accuracy.
• Motion updates are driven by differentiable Gauss-Newton optimization over geometric reprojection errors, while depth estimation relies on a 3D cost volume and learned matching.