[Paper Review] Unpaired Point Cloud Completion on Real Scans using Adversarial Training

Xuelin Chen, Baoquan Chen+1 more|arXiv (Cornell University)|Mar 29, 2019

3D Shape Modeling and Analysis38 references69 citations

TL;DR

This paper presents a point-based GAN framework that completes real-world, partial scans without requiring paired partial-complete data by learning mappings between latent spaces of real partials and clean complete shapes.

ABSTRACT

As 3D scanning solutions become increasingly popular, several deep learning setups have been developed geared towards that task of scan completion, i.e., plausibly filling in regions there were missed in the raw scans. These methods, however, largely rely on supervision in the form of paired training data, i.e., partial scans with corresponding desired completed scans. While these methods have been successfully demonstrated on synthetic data, the approaches cannot be directly used on real scans in absence of suitable paired training data. We develop a first approach that works directly on input point clouds, does not require paired training data, and hence can directly be applied to real scans for scan completion. We evaluate the approach qualitatively on several real-world datasets (ScanNet, Matterport, KITTI), quantitatively on 3D-EPN shape completion benchmark dataset, and demonstrate realistic completions under varying levels of incompleteness.

Motivation & Objective

• Address the lack of paired supervision for real-world scan completion.
• Develop an unpaired point-based completion method operable directly on raw scans.
• Leverage latent-space manifolds to regularize completion across different data distributions.
• Demonstrate robustness and generalization to real and synthetic datasets across multiple object categories.

Proposed method

• Train autoencoders to learn latent spaces for clean-complete point sets and for noisy-partial scans.
• Learn a generator G that maps latent codes from the partial-space X_r to the complete-space X_c.
• Use a discriminator to form a GAN in the latent space, encouraging mapped partials to resemble real complete latent codes.
• Incorporate a Hausdorff-distance based reconstruction loss to ensure partial-consistency between input and completion.
• Adopt a least-squares GAN objective for stabilizing training.
• Operate directly on point clouds with PointNet-like encoders/decoders instead of voxel grids or SDFs.

Experimental results

Research questions

• RQ1 Can unpaired training enable high-quality completion of real-world partial scans without ground-truth correspondences?
• RQ2 Does learning a translation between latent spaces generalize to real vs. synthetic data distributions?
• RQ3 What is the impact of including a Hausdorff-distance reconstruction term on preserving input-consistency in completions?
• RQ4 How does the unpaired method compare to supervised baselines (3D-EPN, PCN) on real and synthetic data?
• RQ5 Can the approach adapt to multiple object categories with limited paired data?

Key findings

• The method enables scan completion directly on real partial scans without paired data and generalizes across ScanNet, Matterport3D, and KITTI.
• On real-world data, the unpaired approach yields more plausible completions than supervised baselines like 3D-EPN and PCN.
• On synthetic data (3D-EPN dataset), the unsupervised method outperforms 3D-EPN and is competitive with PCN; with supervised adaptation, it approaches PCN performance.
• Ablation studies show the GAN component and the Hausdorff reconstruction term are important for achieving best results.