[Paper Review] PF-Net: Point Fractal Network for 3D Point Cloud Completion

Zitian Huang, Yikuan Yu+3 more|arXiv (Cornell University)|Mar 1, 2020

3D Shape Modeling and AnalysisEngineering33 references38 citations

TL;DR

PF-Net preserves the input partial point cloud and hierarchically predicts missing regions with a multi-scale, fractal-like decoder and adversarial loss, achieving high-fidelity 3D point cloud completion.

ABSTRACT

In this paper, we propose a Point Fractal Network (PF-Net), a novel learning-based approach for precise and high-fidelity point cloud completion. Unlike existing point cloud completion networks, which generate the overall shape of the point cloud from the incomplete point cloud and always change existing points and encounter noise and geometrical loss, PF-Net preserves the spatial arrangements of the incomplete point cloud and can figure out the detailed geometrical structure of the missing region(s) in the prediction. To succeed at this task, PF-Net estimates the missing point cloud hierarchically by utilizing a feature-points-based multi-scale generating network. Further, we add up multi-stage completion loss and adversarial loss to generate more realistic missing region(s). The adversarial loss can better tackle multiple modes in the prediction. Our experiments demonstrate the effectiveness of our method for several challenging point cloud completion tasks.

Motivation & Objective

• Motivate robustly repairing incomplete 3D point clouds without altering existing points.
• Develop a hierarchical, multi-scale generator that preserves spatial arrangements while inferring missing geometry.
• Extract rich multi-scale features from partial inputs using a novel multi-resolution encoder.
• Generate missing regions with a Point Pyramid Decoder to reduce genus-wise distortions and preserve details.
• Incorporate multi-stage completion loss and adversarial loss to improve realism and handle multiple output modes.

Proposed method

• Introduce Multi-Resolution Encoder (MRE) with Combined Multi-Layer Perception (CMLP) to extract multi-scale features from the partial point cloud across three scales.
• Use Iterative Farthest Point Sampling (IFPS) to obtain feature points at multiple resolutions for the encoder.
• Design a hierarchical Point Pyramid Decoder (PPD) that outputs missing-region points at three scales via primary, secondary, and detailed point layers guided by feature points.
• Adopt a multi-stage completion loss that compares missing-region predictions at multiple resolutions to ground-truth subsamples of the missing region.
• Train with an adversarial loss where a Discriminator guides PF-Net to produce more realistic missing-region point clouds.
• Combine completion and adversarial losses into a joint objective to balance geometric fidelity and realism.

Experimental results

Research questions

• RQ1 Can partial point clouds be repaired by predicting only the missing region while preserving existing structure?
• RQ2 Does a multi-resolution, feature-point-driven encoder-decoder fully utilize local and global geometry to improve detail in the predicted region?
• RQ3 Can a hierarchical, fractal-like decoder reduce genus-wise distortions and improve detail retention in the missing region?
• RQ4 Does adversarial training improve realism and reduce multi-modal prediction issues in point cloud completion?

Key findings

• PF-Net outperforms baseline methods on both overall completion quality and the missing-region quality across most categories.
• The inclusion of the Discriminator improves prediction quality in the majority of categories.
• CMLP and MR-CMLP enhance feature extraction performance, with PF-Net further improving detail preservation through the PPD decoder.
• The model demonstrates robustness to varying degrees of incomplete input (25%, 50%, 75%) and can handle multiple missing parts.
• Quantitative results show PF-Net and PF-Net (vanilla) achieving lower Pred→GT and GT→Pred errors in most categories and mean across 13 categories.