[Paper Review] DeepSFM: Structure From Motion Via Deep Bundle Adjustment

Xingkui Wei, Yinda Zhang+3 more|arXiv (Cornell University)|Dec 20, 2019

Advanced Vision and Imaging51 references4 citations

TL;DR

DeepSFM proposes a deep learning framework for Structure-from-Motion that iteratively refines 3D depth and camera pose using two differentiable cost volumes: a depth-based cost volume (D-CV) enforcing photo- and geometric consistency, and a pose-based cost volume (P-CV) enforcing consistency through hypothetical camera motion. The method achieves state-of-the-art performance with superior robustness to noisy or sparse initial poses and depth estimates.

ABSTRACT

Structure from motion (SfM) is an essential computer vision problem which has not been well handled by deep learning. One of the promising trends is to apply explicit structural constraint, e.g. 3D cost volume, into the network. However, existing methods usually assume accurate camera poses either from GT or other methods, which is unrealistic in practice. In this work, we design a physical driven architecture, namely DeepSFM, inspired by traditional Bundle Adjustment (BA), which consists of two cost volume based architectures for depth and pose estimation respectively, iteratively running to improve both. The explicit constraints on both depth (structure) and pose (motion), when combined with the learning components, bring the merit from both traditional BA and emerging deep learning technology. Extensive experiments on various datasets show that our model achieves the state-of-the-art performance on both depth and pose estimation with superior robustness against less number of inputs and the noise in initialization.

Motivation & Objective

• Address the limitation of deep learning-based SfM methods that lack explicit geometric constraints and rely on accurate initial camera poses.
• Overcome the brittleness of conventional SfM in textureless or non-Lambertian scenes by integrating physical priors into a deep learning framework.
• Enable robust joint optimization of depth and camera pose even when initial estimates are noisy or sparse, by mimicking the iterative nature of Bundle Adjustment.
• Bridge the gap between traditional geometric SfM and end-to-end deep learning by embedding photo-consistency and geometric-consistency into a learnable, iterative architecture.

Proposed method

• Proposes a physical-driven deep network inspired by Bundle Adjustment, using two distinct differentiable cost volumes: D-CV for depth estimation and P-C- for pose refinement.
• D-CV computes photometric and geometric consistency costs by unprojecting pixels into discrete depth planes and measuring differences in image features and depth values across views.
• P-CV performs a similar cost computation but by hypothetically moving the camera to discrete positions, measuring photometric and geometric differences to refine pose.
• The network alternates between D-CV and P-CV inference in iterative refinement steps, improving both depth and pose estimates in a coordinate-descent-like manner.
• Initial depth is obtained via conventional 3D cost volume; initial pose is derived from a direct regression model like DeMoN, enabling end-to-end training from weak supervision.
• The iterative process avoids the memory and computational bottlenecks of unrolling LM optimization, enabling more iterations and better convergence than prior methods.

Experimental results

Research questions

• RQ1 Can a deep learning framework effectively embed explicit geometric constraints—photo-consistency and geometric-consistency—into the network for joint SfM optimization?
• RQ2 How does the performance of a deep learning-based SfM method compare to conventional Bundle Adjustment when initialized with noisy or inaccurate camera poses?
• RQ3 To what extent can iterative refinement via D-CV and P-CV improve both depth and pose estimation beyond a single forward pass?
• RQ4 Does the proposed method maintain robustness when the number of input views is reduced or when input poses are corrupted by noise?
• RQ5 Can the integration of physical priors into a differentiable architecture outperform end-to-end regression-based SfM models in terms of generalization and accuracy?

Key findings

• DeepSFM achieves state-of-the-art performance on DeMoN, ScanNet, ETH3D, and Tanks and Temples datasets, outperforming both conventional SfM (e.g., COLMAP) and SOTA deep learning methods.
• On ETH3D, DeepSFM reduces depth error by half compared to COLMAP under the same number of views, or achieves similar error with only half the number of input images.
• When input pose quality degrades via image downscaling or frame sub-sampling, DeepSFM maintains stable performance, while COLMAP and R-MVSNet show significant performance drops.
• The iterative refinement process converges after approximately four iterations, with both depth and pose errors consistently decreasing, demonstrating effective mutual improvement.
• Ablation studies show that the P-CV module significantly improves pose estimation over direct visual odometry baselines, especially in early iterations with noisy depth.
• Visualizations of camera trajectories and 3D reconstructions confirm higher-quality and more stable outputs, even in challenging scenes with low texture or reflective surfaces.