[Paper Review] Semantic Photometric Bundle Adjustment on Natural Sequences

Rui Zhu, Chaoyang Wang+3 more|arXiv (Cornell University)|Nov 30, 2017

Robotics and Sensor-Based Localization22 references3 citations

TL;DR

This paper proposes semantic photometric bundle adjustment (PBA) that integrates a deep learning-based 3D shape prior into a photometric optimization framework to enable dense, accurate 3D reconstruction of objects from natural image sequences. By leveraging semantic priors, the method achieves state-of-the-art performance in depth accuracy and reconstruction density, even under low-texture conditions or limited viewpoint coverage, while maintaining SLAM-like camera pose estimation with PBA-level photometric consistency.

ABSTRACT

The problem of obtaining dense reconstruction of an object in a natural sequence of images has been long studied in computer vision. Classically this problem has been solved through the application of bundle adjustment (BA). More recently, excellent results have been attained through the application of photometric bundle adjustment (PBA) methods -- which directly minimize the photometric error across frames. A fundamental drawback to BA & PBA, however, is: (i) their reliance on having to view all points on the object, and (ii) for the object surface to be well textured. To circumvent these limitations we propose semantic PBA which incorporates a 3D object prior, obtained through deep learning, within the photometric bundle adjustment problem. We demonstrate state of the art performance in comparison to leading methods for object reconstruction across numerous natural sequences.

Motivation & Objective

• To address the limitations of classical and photometric bundle adjustment (PBA) in reconstructing dense 3D shapes from natural sequences when objects are poorly textured or only partially observed.
• To integrate a learned 3D shape prior from deep learning into the PBA optimization pipeline to improve reconstruction density and robustness.
• To enable accurate, globally consistent 6DoF camera pose estimation and dense depth map recovery in natural sequences without requiring extensive viewpoint coverage or high texture.
• To bridge the gap between deep learning-based 3D reconstruction and geometric optimization by enforcing photometric consistency at test time.

Proposed method

• The method formulates a semantic PBA objective that jointly optimizes camera poses and depth maps while incorporating a 3D shape prior derived from a deep neural network.
• It uses a learned shape prior (e.g., from ShapeNet) as a geometric prior to regularize the optimization, enabling reconstruction even when points are self-occluded or poorly textured.
• The optimization minimizes photometric error across frames, reprojecting the estimated 3D shape into image planes using estimated camera poses to enforce consistency.
• The method initializes with results from classic PBA (e.g., from Ham et al. [11]) as a strong baseline, improving convergence and accuracy.
• Silhouette and depth constraints are enforced during optimization to further regularize the solution.
• A new dataset is collected with both rendered and natural sequences, including ground truth camera poses, depth maps, and CAD-aligned shapes for evaluation.

Experimental results

Research questions

• RQ1 Can a learned 3D shape prior significantly improve the density and accuracy of 3D reconstruction in natural image sequences where classical PBA fails due to poor texture or limited visibility?
• RQ2 How does integrating semantic priors into the PBA optimization pipeline affect convergence robustness under large initialization errors in camera pose?
• RQ3 To what extent can semantic PBA outperform both classic PBA and deep learning-based methods in terms of depth accuracy and reconstruction density on real-world natural sequences?
• RQ4 Does enforcing photometric consistency at test time, combined with a shape prior, lead to more reliable and geometrically consistent 3D reconstructions than end-to-end learning methods?

Key findings

• On rendered sequences, the proposed method achieves a depth error of 0.0627 and a density of 0.9342, outperforming Ham et al. (0.0682 depth error, 0.4732 density) and openMVS (0.0731 depth error, 0.5862 density).
• On natural sequences, the method achieves a depth error of 0.0756 and a density of 0.9076, significantly outperforming openMVS (0.0798 depth error, 0.6987 density) and Ham et al. (0.0804 depth error, 0.6558 density).
• The method demonstrates superior robustness to initialization error, converging reliably even with large initial pose errors, as shown in Figure 8, where it outperforms Zhu et al. [26] in convergence stability.
• The method achieves SLAM-like camera pose estimation with PBA-level accuracy while producing denser depth maps than both openMVS and Ham et al., thanks to the semantic shape prior.
• Qualitative results show that the method produces full 3D shapes and globally consistent camera trajectories even from as few as two frames, while classic PBA methods fail under such low-observation conditions.
• Learning-based methods trained on synthetic data (e.g., [3, 6]) suffer from domain gap issues on natural sequences, whereas the proposed method maintains strong performance due to geometric consistency enforcement at test time.