[Paper Review] REGTR: End-to-end Point Cloud Correspondences with Transformers

Zi Jian Yew, Gim Hee Lee|arXiv (Cornell University)|Mar 28, 2022

3D Shape Modeling and Analysis17 citations

TL;DR

REGTR proposes an end-to-end point cloud registration framework using transformers to directly predict clean, consistent correspondences without relying on nearest-neighbor matching or RANSAC. By leveraging multi-head self- and cross-attention mechanisms with positional encoding, it achieves state-of-the-art performance on 3DMatch and ModelNet benchmarks with faster inference and improved robustness.

ABSTRACT

Despite recent success in incorporating learning into point cloud registration, many works focus on learning feature descriptors and continue to rely on nearest-neighbor feature matching and outlier filtering through RANSAC to obtain the final set of correspondences for pose estimation. In this work, we conjecture that attention mechanisms can replace the role of explicit feature matching and RANSAC, and thus propose an end-to-end framework to directly predict the final set of correspondences. We use a network architecture consisting primarily of transformer layers containing self and cross attentions, and train it to predict the probability each point lies in the overlapping region and its corresponding position in the other point cloud. The required rigid transformation can then be estimated directly from the predicted correspondences without further post-processing. Despite its simplicity, our approach achieves state-of-the-art performance on 3DMatch and ModelNet benchmarks. Our source code can be found at https://github.com/yewzijian/RegTR .

Motivation & Objective

• To eliminate the need for post-processing steps like RANSAC and nearest-neighbor feature matching in point cloud registration.
• To investigate whether attention mechanisms can directly predict final correspondences instead of learning discriminative features.
• To improve registration accuracy and efficiency by training a model to output clean, rigid-consistent correspondences end-to-end.
• To reduce dependency on high-resolution keypoint sampling and feature engineering by using grid-subsampled points with attention-based correspondence prediction.
• To demonstrate that direct correspondence prediction via transformers outperforms conventional pipelines combining learned features with RANSAC.

Proposed method

• Uses a point convolutional backbone to downsample input point clouds and extract initial features.
• Employs multiple transformer layers with multi-head self- and cross-attention to aggregate global context and enforce rigidity constraints via positional encoding.
• Predicts for each downsampled point: (1) its corresponding location in the other point cloud, and (2) an overlap probability score.
• Estimates the final rigid transformation directly from the predicted correspondences using a differentiable least-squares solver.
• Trains the network using a combination of overlap loss ($\mathcal{L}_o$), feature loss ($\mathcal{L}_f$), and a contrastive loss (Circle Loss) on the final transformer layer.
• Avoids supervision on intermediate layers to prevent performance degradation, applying losses only on the final output layer.

Experimental results

Research questions

• RQ1 Can self- and cross-attention mechanisms in transformers directly predict final point correspondences without explicit feature matching?
• RQ2 Does end-to-end training of a transformer-based network for correspondence prediction lead to better registration accuracy than traditional pipelines with RANSAC?
• RQ3 How does the performance of direct correspondence prediction compare to methods relying on nearest-neighbor matching and RANSAC on standard benchmarks?
• RQ4 What is the impact of different loss functions and supervision strategies on the quality of predicted correspondences?
• RQ5 Can a simpler, direct prediction approach outperform complex feature learning pipelines when using downsampled, grid-based points?

Key findings

• REGTR achieves a registration recall of 92.0% on the 3DMatch benchmark, surpassing the previous SOTA and outperforming RANSAC-based baselines.
• On the 3DLoMatch dataset, REGTR achieves 64.8% registration recall, demonstrating strong generalization to challenging, low-overlap scenes.
• The method reduces rotation error to 1.567° and translation error to 0.049m on 3DMatch, outperforming both RANSAC baselines and feature-matching approaches.
• Ablation studies show that removing the feature loss reduces registration recall by 1.6% on 3DMatch, indicating its importance for accurate correspondence prediction.
• Applying losses on all transformer layers instead of just the final layer degrades performance by 8.1% on 3DMatch, confirming that final-layer supervision is optimal.
• Applying RANSAC to REGTR’s predicted correspondences yields marginally worse results, confirming that the predicted correspondences are already consistent and robust.