[Paper Review] Learning Multiview 3D Point Cloud Registration

Žan Gojčič, Caifa Zhou+3 more|arXiv (Cornell University)|Jan 1, 2020

3D Shape Modeling and Analysis70 references12 citations

TL;DR

This paper proposes the first end-to-end, differentiable deep learning framework for multiview 3D point cloud registration, jointly optimizing pairwise alignment and global synchronization via a differentiable Procrustes solver and spectral relaxation. It achieves state-of-the-art performance with a 25+ percentage point improvement in rotation error and over 13x speedup over RANSAC-based methods on 60-scan scenes, using a novel confidence estimation block and iterative reweighted least squares refinement.

ABSTRACT

We present a novel, end-to-end learnable, multiview 3D point cloud registration algorithm. Registration of multiple scans typically follows a two-stage pipeline: the initial pairwise alignment and the globally consistent refinement. The former is often ambiguous due to the low overlap of neighboring point clouds, symmetries and repetitive scene parts. Therefore, the latter global refinement aims at establishing the cyclic consistency across multiple scans and helps in resolving the ambiguous cases. In this paper we propose, to the best of our knowledge, the first end-to-end algorithm for joint learning of both parts of this two-stage problem. Experimental evaluation on well accepted benchmark datasets shows that our approach outperforms the state-of-the-art by a significant margin, while being end-to-end trainable and computationally less costly. Moreover, we present detailed analysis and an ablation study that validate the novel components of our approach. The source code and pretrained models are publicly available under https://github.com/zgojcic/3D_multiview_reg.

Motivation & Objective

• To address the limitations of traditional two-stage 3D point cloud registration pipelines, which suffer from error propagation and lack of global consistency.
• To eliminate the need for separate, non-differentiable global refinement steps by jointly learning pairwise and global alignment in an end-to-end differentiable manner.
• To improve robustness and accuracy in low-overlap, symmetric, or repetitive scenes by leveraging iterative feedback between local and global optimization.
• To develop a computationally efficient, fully differentiable registration framework that operates directly on point clouds without intermediate representations like depth images.

Proposed method

• Introduces a differentiable pairwise registration block that estimates transformation parameters via a differentiable Procrustes solver, enabling backpropagation through the alignment process.
• Employs a novel confidence estimation block with a learnable overlap pooling layer to predict reliability scores for pairwise matches.
• Uses a differentiable spectral relaxation layer for transformation synchronization, enabling global optimization of pose graphs with learnable edge weights.
• Applies an iterative reweighted least squares (IRLS) scheme to refine both pairwise and absolute transformations by feeding residuals back through the network.
• Introduces an edge pruning mechanism based on predicted confidence scores, dynamically removing low-confidence edges during optimization.
• End-to-end training is achieved by composing the pairwise network, confidence head, and synchronization layer into a single differentiable computational graph.

Experimental results

Research questions

• RQ1 Can a fully end-to-end deep learning framework jointly optimize pairwise and global 3D point cloud registration, eliminating the need for separate, non-differentiable refinement stages?
• RQ2 How does iterative feedback between local pairwise estimates and global synchronization improve registration accuracy, especially in ambiguous or low-overlap scenarios?
• RQ3 To what extent does a learnable confidence estimation block improve robustness to outliers and noisy initial matches compared to fixed or heuristic weighting schemes?
• RQ4 What is the impact of differentiable optimization layers (e.g., Procrustes and spectral relaxation) on convergence and performance in multiview registration?
• RQ5 How does the proposed method compare in speed and accuracy to RANSAC-based and other state-of-the-art multiview registration pipelines?

Key findings

• The proposed method achieves a 25.3 percentage point average improvement in rotation error over the state-of-the-art on the 3DMatch benchmark, significantly outperforming existing methods.
• The method is over 13 times faster than RANSAC-based pipelines, completing registration of a 60-scan Kitchen scene in approximately 80 seconds compared to over 1100 seconds.
• The iterative reweighting scheme improves pairwise transformation estimates by approximately 2 percentage points in recall over the initial estimate after four iterations.
• The confidence estimation block with overlap pooling provides a >20 percentage point gain in performance over alternative pruning strategies, demonstrating its critical role in robustness.
• The method generalizes better to unseen scenes, achieving 4 percentage points higher recall on the Redwood indoor dataset compared to the state-of-the-art.
• Ablation studies confirm that both the confidence-based edge pruning and iterative refinement are essential for optimal performance, with removal of either component leading to significant performance degradation.