[Paper Review] Extreme Relative Pose Network under Hybrid Representations

Zhenpei Yang, Siming Yan+1 more|arXiv (Cornell University)|Dec 25, 2019

Robotics and Sensor-Based Localization47 references4 citations

TL;DR

This paper proposes a novel RGB-D based relative pose estimation method for small- or non-overlapping scans using hybrid representations (360-image, 2D layout, and planar patches) to enable adaptive feature extraction and scene completion. It employs a global-to-local matching pipeline that refines initial poses using geometric constraints, achieving state-of-the-art performance with top-1 rotation/translation errors of 28.6°/0.90m on ScanNet and significantly improving few-view 3D reconstruction quality.

ABSTRACT

In this paper, we introduce a novel RGB-D based relative pose estimation approach that is suitable for small-overlapping or non-overlapping scans and can output multiple relative poses. Our method performs scene completion and matches the completed scans. However, instead of using a fixed representation for completion, the key idea is to utilize hybrid representations that combine 360-image, 2D image-based layout, and planar patches. This approach offers adaptively feature representations for relative pose estimation. Besides, we introduce a global-2-local matching procedure, which utilizes initial relative poses obtained during the global phase to detect and then integrate geometric relations for pose refinement. Experimental results justify the potential of this approach across a wide range of benchmark datasets. For example, on ScanNet, the rotation translation errors of the top-1/top-5 predictions of our approach are 28.6/0.90m and 16.8/0.76m, respectively. Our approach also considerably boosts the performance of multi-scan reconstruction in few-view reconstruction settings.

Motivation & Objective

• To address the challenge of relative pose estimation between RGB-D scans with minimal or no overlap, where traditional feature matching fails due to insufficient overlap.
• To improve robustness and accuracy in extreme relative pose estimation by leveraging multiple complementary 3D representations for scan completion.
• To enable multi-hypothesis output for ambiguous pose estimation scenarios, enhancing reliability in real-world robotics and reconstruction applications.
• To integrate geometric constraints (e.g., parallelism, coplanarity) in a local refinement stage to improve pose accuracy after global initialization.

Proposed method

• Uses a hybrid representation combining 360-image, 2D layout, and planar patches to complete incomplete or non-overlapping scans, enabling richer and adaptive feature learning.
• Applies a global matching module using spectral matching on features from completed scans to generate initial relative pose candidates.
• Employs a local refinement module that detects and integrates geometric relations (e.g., parallel planes, distances) using robust norms to refine initial poses.
• Introduces a sequential global-to-local pipeline that allows iterative refinement and supports multiple output poses to model ambiguity.
• Combines the method with MRF-SFM’s pose optimizer to enable consistent multi-scan alignment from few views.
• Uses perturbed scan pairs to evaluate geometric relation prediction accuracy, with robust filtering to handle imperfect predictions.

Experimental results

Research questions

• RQ1 Can hybrid 3D representations (360-image, 2D layout, planar patches) improve relative pose estimation in non-overlapping scans compared to single-representation methods?
• RQ2 How effective is a global-to-local matching pipeline in refining relative poses using geometric constraints when initial poses are uncertain?
• RQ3 To what extent can multiple relative pose predictions improve the quality of multi-scan 3D reconstruction in few-view settings?
• RQ4 How does the integration of predicted geometric relations (e.g., parallelism, coplanarity) enhance pose refinement in the absence of strong overlap?
• RQ5 Can the proposed method outperform state-of-the-art approaches in both overlapping and non-overlapping scan scenarios?

Key findings

• On ScanNet, the proposed method achieves top-1 rotation/translation errors of 28.6°/0.90m, outperforming the SOTA baseline of 34.1°/0.93m.
• For top-5 predictions, the errors drop to 16.8°/0.76m on ScanNet, demonstrating the benefit of multi-hypothesis output in handling pose ambiguity.
• On SUNCG, the method reduces top-1 errors to 18.1°/0.16m, compared to 31.1°/0.27m for the SOTA baseline.
• The local refinement module reduces errors by 15–20% across datasets, showing its effectiveness in improving initial global estimates.
• In few-view reconstruction (5 scans), the method with top-5 outputs achieves 1.81°/0.09m mean error, outperforming RobustRecons (12.3°/0.26m) and Fine-to-Coarse (68.6°/1.43m) in sparse settings.
• Geometric relation prediction achieves 93% accuracy for co-planar relations, and robust filtering enables high performance despite imperfect predictions.