[Paper Review] PRNet: Self-Supervised Learning for Partial-to-Partial Registration

Yue Wang, Justin Solomon|arXiv (Cornell University)|Oct 27, 2019

3D Shape Modeling and Analysis112 citations

TL;DR

PRNet introduces a self-supervised, iterative framework for partial-to-partial point cloud registration that learns keypoints, correspondences, and rigid transforms via Gumbel–Softmax and an actor–critic style module, achieving state-of-the-art results on synthetic and real data while enabling transfer to classification.

ABSTRACT

We present a simple, flexible, and general framework titled Partial Registration Network (PRNet), for partial-to-partial point cloud registration. Inspired by recently-proposed learning-based methods for registration, we use deep networks to tackle non-convexity of the alignment and partial correspondence problems. While previous learning-based methods assume the entire shape is visible, PRNet is suitable for partial-to-partial registration, outperforming PointNetLK, DCP, and non-learning methods on synthetic data. PRNet is self-supervised, jointly learning an appropriate geometric representation, a keypoint detector that finds points in common between partial views, and keypoint-to-keypoint correspondences. We show PRNet predicts keypoints and correspondences consistently across views and objects. Furthermore, the learned representation is transferable to classification.

Motivation & Objective

• Address partial-to-partial point cloud registration where only subsets of the shapes are visible.
• Develop a self-supervised framework to learn geometric representations, keypoint detectors, and keypoint correspondences without labeled data.
• Enable iterative coarse-to-fine refinement of alignments akin to ICP while leveraging learning-based robustness.
• Demonstrate transferability of learned representations to 3D shape classification.
• Provide a reproducible approach with code release for researchers in 3D vision and registration.

Proposed method

• Use a deep network to learn co-contextual embeddings for two point clouds via DGCNN and Transformer.
• Detect mutually shared keypoints by leveraging the L2 norms of learned features to select top-k points.
• Predict keypoint-to-keypoint correspondences with a Gumbel–Softmax sampler and a straight-through gradient estimator.
• Incorporate an adaptive temperature (lambda) predicted by a critic-like network to control the sharpness of the correspondence mapping.
• Solve the Procrustes alignment on the detected keypoints to obtain rigid transformation estimates.
• Iteratively apply PRNet to refine alignment, updating embeddings and keypoints at each step.

Experimental results

Research questions

• RQ1 Can PRNet robustly perform partial-to-partial registration in the absence of full-view correspondence?
• RQ2 Does self-supervised learning of keypoint detection and correspondences yield transferable representations for downstream tasks like classification?
• RQ3 Can adaptive, actor–critic–inspired control of mapping sharpness improve registration accuracy across varying viewing conditions?
• RQ4 How does PRNet compare to classical and other learning-based registration methods on synthetic and real data in partial-to-partial settings?

Key findings

• PRNet achieves state-of-the-art performance among tested methods for partial-to-partial registration on synthetic data (ModelNet40) and real data, outperforming ICP, Go-ICP, FGR, PointNetLK, and DCP-v2 in several metrics.
• The method’s learned keypoints and correspondences remain consistent across views and objects, enabling reliable Procrustes alignment.
• Adaptive Gumbel–Softmax with a predicted temperature (lambda) improves the sharpness of correspondences when appropriate, yielding more accurate rigid transformations.
• Registration representations learned by PRNet transfer to 3D shape classification with competitive accuracy using a linear SVM on ShapeNetCore-derived embeddings.
• Inference-time fine-tuning on real scans (for improved realism) demonstrates practical applicability and robustness to real-world data.