[Paper Review] PoseDiffusion: Solving Pose Estimation via Diffusion-aided Bundle Adjustment

Jianyuan Wang, Christian Rupprecht+1 more|arXiv (Cornell University)|Jun 27, 2023

Advanced Vision and ImagingComputer Science3 citations

TL;DR

PoseDiffusion proposes a diffusion-based framework for camera pose estimation that integrates geometric constraints from 2D correspondences into a probabilistic diffusion process to iteratively refine camera poses. It achieves state-of-the-art accuracy on CO3Dv2 and RealEstate10k, outperforming classical SfM and learned methods, especially in sparse-view and challenging settings.

ABSTRACT

Camera pose estimation is a long-standing computer vision problem that to date often relies on classical methods, such as handcrafted keypoint matching, RANSAC and bundle adjustment. In this paper, we propose to formulate the Structure from Motion (SfM) problem inside a probabilistic diffusion framework, modelling the conditional distribution of camera poses given input images. This novel view of an old problem has several advantages. (i) The nature of the diffusion framework mirrors the iterative procedure of bundle adjustment. (ii) The formulation allows a seamless integration of geometric constraints from epipolar geometry. (iii) It excels in typically difficult scenarios such as sparse views with wide baselines. (iv) The method can predict intrinsics and extrinsics for an arbitrary amount of images. We demonstrate that our method PoseDiffusion significantly improves over the classic SfM pipelines and the learned approaches on two real-world datasets. Finally, it is observed that our method can generalize across datasets without further training. Project page: https://posediffusion.github.io/

Motivation & Objective

• To address the limitations of classical Structure-from-Motion (SfM) pipelines in sparse-view and wide-baseline scenarios.
• To integrate geometric constraints from epipolar geometry into a deep learning framework for improved pose estimation.
• To develop a differentiable, iterative refinement process that mimics bundle adjustment but with the inductive bias of diffusion models.
• To enable end-to-end prediction of both extrinsic and intrinsic camera parameters from free-form image collections.

Proposed method

• Formulates camera pose estimation as learning the conditional distribution p(x|I) using a diffusion model, where x is the camera pose and I is the input image set.
• Uses a denoising U-Net architecture to iteratively refine noisy camera pose estimates by reversing a forward diffusion process.
• Incorporates geometric guidance via 2D image-to-image correspondences to enforce epipolar constraints during sampling, improving geometric consistency.
• Employs a multi-scale feature extractor (e.g., DINO ViT-S16) to generate image embeddings used as conditioning for the diffusion model.
• Applies classifier-free guidance with geometric constraints to steer the sampling toward more consistent and accurate pose solutions.
• Trains the model in a self-supervised manner on large-scale datasets with known camera poses, enabling generalization to unseen scenes.

Experimental results

Research questions

• RQ1 Can a diffusion-based framework effectively model the complex, non-convex optimization landscape of bundle adjustment in camera pose estimation?
• RQ2 How does integrating geometric constraints from 2D correspondences improve the accuracy and consistency of diffusion-based pose estimation?
• RQ3 Can a learned diffusion model generalize across datasets and scene types without fine-tuning, especially in low-data regimes?
• RQ4 Does the method outperform classical SfM pipelines and learned baselines in both sparse and dense view settings?
• RQ5 To what extent does background context contribute to pose estimation accuracy in symmetric or texture-poor scenes?

Key findings

• PoseDiffusion achieves 66.5 mAA(30) on CO3Dv2 using a DINO ViT-S16 backbone, outperforming ResNet50 and supervised ResNet50.
• The method performs on par with COLMAP+SPSG in the many-frame regime, despite COLMAP having access to ground-truth annotations.
• Using 100 diffusion steps (T=100) yields optimal performance, with no significant gain beyond 100 steps.
• Masking out background pixels reduces mAA(30) to 57.0, indicating that background context significantly improves pose estimation.
• The model generalizes across datasets without fine-tuning, demonstrating strong zero-shot generalization capability.
• PoseDiffusion improves NeRF training supervision, showing superior accuracy in both extrinsic and intrinsic camera parameter estimation.