[Paper Review] 3D Point Cloud Registration with Multi-Scale Architecture and Unsupervised Transfer Learning

Sofiane Horache, Jean‐Emmanuel Deschaud+1 more|arXiv (Cornell University)|Mar 26, 2021

3D Shape Modeling and Analysis55 references29 citations

TL;DR

The paper introduces MS-SVConv, a multi-scale U-Net for fast, generalizable 3D point cloud registration, and UDGE, an unsupervised transfer-learning approach to adapt to new datasets.

ABSTRACT

We propose a method for generalizing deep learning for 3D point cloud registration on new, totally different datasets. It is based on two components, MS-SVConv and UDGE. Using Multi-Scale Sparse Voxel Convolution, MS-SVConv is a fast deep neural network that outputs the descriptors from point clouds for 3D registration between two scenes. UDGE is an algorithm for transferring deep networks on unknown datasets in a unsupervised way. The interest of the proposed method appears while using the two components, MS-SVConv and UDGE, together as a whole, which leads to state-of-the-art results on real world registration datasets such as 3DMatch, ETH and TUM. The code is publicly available at https://github.com/humanpose1/MS-SVConv .

Motivation & Objective

• Address the challenge of generalizing deep learning-based 3D point cloud registration to new, diverse datasets without requiring ground-truth poses.
• Propose a fast, generalizable descriptor computation architecture using multi-scale sparse voxel convolutions (MS-SVConv).
• Introduce UDGE, an unsupervised data-generation-based transfer learning strategy to adapt pre-trained models to unseen datasets.
• Demonstrate state-of-the-art or strongly competitive results on real-world datasets (3DMatch, ETH, TUM) under supervised and unsupervised transfer settings.

Proposed method

• Propose MS-SVConv: a shared multi-scale U-Net-based descriptor extractor operating on point clouds at multiple voxel scales, with scale-wise downsampling and a final MLP fusion to produce point descriptors.
• Use sparse convolutions to maintain efficiency and enable handling of large point clouds.
• Train with a hard negative contrastive loss to learn discriminative descriptors (positive and negative matches).
• Introduce UDGE: data-generation-based unsupervised transfer learning that creates two partial views from a single cloud (via crop and periodic sampling) to enable transfer to target datasets without ground-truth poses.
• During transfer, pre-train on a source dataset (e.g., ModelNet or 3DMatch) and apply UDGE on the target dataset to adapt descriptors without supervision.

Figure 1 : Summary of the proposed method. We can use MS-SVConv for supervised training on a huge dataset, and then we can transfer it in an unsupervised fashion on a smaller dataset.

Experimental results

Research questions

• RQ1 Can a multi-scale U-Net descriptor architecture improve registration generalization across diverse real-world datasets?
• RQ2 Does unsupervised transfer learning via data generation (UDGE) enable effective adaptation of pre-trained descriptors to unseen datasets without ground-truth poses?
• RQ3 What is the performance gain from using multi-scale MS-SVConv alone vs. in combination with UDGE on challenging datasets (ETH, TUM, 3DMatch)?

Key findings

• MS-SVConv with three heads (three scales) outperforms published methods on 3DMatch in FMR, achieving 98.4% vs. 97.5% for the best prior method under one test setup.
• Multi-scale MS-SVConv significantly improves generalization over a single-scale variant, especially when symmetric testing is applied.
• UDO-based transfer (UAGE) on ETH yields state-of-the-art FMR (e.g., 98.9% with MS-SVConv(3) after UDGE) and competitive SRE, while being substantially faster than patch-based methods.
• Pre-training MS-SVConv on a synthetic data source (ModelNet) plus UDGE achieves strong results on ETH, TUM, and 3DMatch without target-ground-truth poses.
• There is a synergy between multi-scale architecture and UDGE, with additive gains in FMR when using more heads (MS-SVConv(3) vs. MS-SVConv(1)).
• UDA-based transfer remains effective even when the target dataset is small, provided there is supervised pre-training on a synthetic or different real-world source.

Figure 2 : Architecture of the proposed Multi-Scale Sparse Voxel Convolution (MS-SVConv) for registration. The point cloud is downsampled at each scale with a grid subsampling that increases the side length of the voxel by 2 each time.