[Paper Review] PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers

Xumin Yu, Yongming Rao+4 more|arXiv (Cornell University)|Aug 19, 2021

3D Shape Modeling and AnalysisEngineering45 references25 citations

TL;DR

PoinTr reframes point cloud completion as set-to-set translation using a geometry-aware Transformer with point proxies, achieving state-of-the-art results on diverse benchmarks including ShapeNet-55/34, PCN, and KITTI.

ABSTRACT

Point clouds captured in real-world applications are often incomplete due to the limited sensor resolution, single viewpoint, and occlusion. Therefore, recovering the complete point clouds from partial ones becomes an indispensable task in many practical applications. In this paper, we present a new method that reformulates point cloud completion as a set-to-set translation problem and design a new model, called PoinTr that adopts a transformer encoder-decoder architecture for point cloud completion. By representing the point cloud as a set of unordered groups of points with position embeddings, we convert the point cloud to a sequence of point proxies and employ the transformers for point cloud generation. To facilitate transformers to better leverage the inductive bias about 3D geometric structures of point clouds, we further devise a geometry-aware block that models the local geometric relationships explicitly. The migration of transformers enables our model to better learn structural knowledge and preserve detailed information for point cloud completion. Furthermore, we propose two more challenging benchmarks with more diverse incomplete point clouds that can better reflect the real-world scenarios to promote future research. Experimental results show that our method outperforms state-of-the-art methods by a large margin on both the new benchmarks and the existing ones. Code is available at https://github.com/yuxumin/PoinTr

Motivation & Objective

• Motivate and address incomplete real-world point clouds due to occlusion, limited sensors, and viewpoints.
• Reformulate point cloud completion as a set-to-set translation task.
• Leverage a Transformer encoder-decoder with geometry-aware blocks to learn structural 3D relationships.
• Develop new challenging benchmarks to better reflect real-world incompleteness and diversity.

Proposed method

• Represent partial point clouds as a sequence of point proxies feeding a Transformer encoder-decoder.
• Introduce a geometry-aware Transformer block that explicitly models local 3D geometric relations via a kNN-based approach.
• Use dynamic, encoder-conditioned queries in the decoder to generate missing point proxies.
• Employ a multi-scale generation module that reconstructs missing parts in a coarse-to-fine manner using FoldingNet.
• Optimize with Chamfer Distance losses on both proxy centers and full point clouds (J = J0 + J1).

Experimental results

Research questions

• RQ1 Can transformers effectively model long-range and local geometric dependencies for point cloud completion?
• RQ2 Does introducing geometry-aware blocks and dynamic queries improve completion quality and generalization to diverse shapes and incompleteness levels?
• RQ3 Do new diverse benchmarks better reveal model robustness and generalization in real-world scenarios?
• RQ4 How does PoinTr perform relative to prior state-of-the-art methods across diverse categories and viewpoints?
• RQ5 Can the approach generalize from ShapeNet-based benchmarks to real-world LiDAR data (KITTI) after fine-tuning?

Key findings

• PoinTr outperforms prior state-of-the-art methods by a large margin on multiple benchmarks, including ShapeNet-55, ShapeNet-34, PCN, and KITTI.
• On ShapeNet-55, PoinTr achieves substantial CD-ell2 improvements over GRNet (0.58 in Simple, 0.60 in Moderate, 0.69 in Hard settings).
• On ShapeNet-55, PoinTr attains 0.46 F-Score versus 0.24 for the previous best (GRNet).
• On ShapeNet-34 seen categories, PoinTr leads all methods; it also generalizes well to unseen categories, with smaller performance degradation in simpler settings.
• On PCN, PoinTr achieves an 8.38 CD-ell1 average across categories, outperforming FoldingNet, PCN, TopNet, PFNet, GRNet, and others.
• Qualitative results show PoinTr recovering finer details and boundaries (e.g., car tires) and being more robust to various incomplete patterns.