[Paper Review] Learning Local Displacements for Point Cloud Completion

Yida Wang, David Joseph Tan+2 more|arXiv (Cornell University)|Mar 30, 2022

3D Shape Modeling and Analysis6 citations

TL;DR

This paper proposes a novel point cloud completion framework using three custom layers—local descriptor matching, neighbor-pooling, and feature-aware up-sampling—within encoder-decoder and transformer architectures. The method achieves state-of-the-art performance on both object and scene completion benchmarks, with a 3.04 Chamfer distance on CompleteScanNet and 4.38 on NYU, demonstrating superior reconstruction quality and fine-grained detail preservation.

ABSTRACT

We propose a novel approach aimed at object and semantic scene completion from a partial scan represented as a 3D point cloud. Our architecture relies on three novel layers that are used successively within an encoder-decoder structure and specifically developed for the task at hand. The first one carries out feature extraction by matching the point features to a set of pre-trained local descriptors. Then, to avoid losing individual descriptors as part of standard operations such as max-pooling, we propose an alternative neighbor-pooling operation that relies on adopting the feature vectors with the highest activations. Finally, up-sampling in the decoder modifies our feature extraction in order to increase the output dimension. While this model is already able to achieve competitive results with the state of the art, we further propose a way to increase the versatility of our approach to process point clouds. To this aim, we introduce a second model that assembles our layers within a transformer architecture. We evaluate both architectures on object and indoor scene completion tasks, achieving state-of-the-art performance.

Motivation & Objective

• Address the challenge of 3D shape completion from partial point clouds, particularly focusing on recovering occluded regions with high geometric fidelity.
• Overcome limitations in existing point cloud processing, such as loss of individual feature descriptors during pooling and unstructured feature maps.
• Improve completion quality by leveraging learned local displacements and adaptive feature aggregation in both encoder-decoder and transformer-based architectures.
• Enable versatile, high-resolution reconstruction for both object and semantic scene completion tasks using a unified, learnable framework.

Proposed method

• Introduces a feature extraction layer that matches point features to a set of pre-trained local descriptors to enhance geometric representation.
• Proposes neighbor-pooling—a novel down-sampling operation that preserves individual feature vectors by selecting those with highest activations, avoiding descriptor loss from max-pooling.
• Designs a feature-aware up-sampling mechanism in the decoder that increases output dimension while maintaining spatial and semantic coherence.
• Integrates the proposed layers into a transformer-based architecture inspired by PoinTr, enabling long-range dependency modeling and improved feature refinement.
• Employs a progressive loss function that reconstructs the target from observable to occluded regions, enhancing structural accuracy.
• Utilizes a mix-and-match ablation study to validate the modularity and effectiveness of each component in the backbone and coarse-to-fine refinement pipeline.

Experimental results

Research questions

• RQ1 How can local feature matching and adaptive pooling improve point cloud completion beyond standard max-pooling and global feature aggregation?
• RQ2 To what extent do the proposed layers enhance reconstruction quality and fine-grained detail recovery in partial point cloud completion?
• RQ3 Can the integration of these layers into a transformer architecture further improve performance on both object and scene completion tasks?
• RQ4 How does the progressive loss function that prioritizes occluded regions affect the final reconstruction quality compared to standard L2 losses?

Key findings

• The proposed method achieves a Chamfer distance of 3.04 on the CompleteScanNet dataset, outperforming prior state-of-the-art methods such as PoinTr (5.08) and MSN (7.28).
• On the NYU dataset, the method achieves a Chamfer distance of 4.38, significantly outperforming GRNet (5.80), VRCNet (5.45), and PoinTr (5.92).
• The ablation study confirms that the neighbor-pooling layer reduces descriptor loss and improves feature representation, contributing to better reconstruction.
• The transformer-based architecture with the proposed components achieves the best results across all benchmarks, demonstrating the effectiveness of the modular design.
• The adaptive weighting of the semantic loss function (via γ) improves IoU by 7.5–9% when removed, proving its importance in balancing semantic and geometric accuracy.
• The mix-and-match ablation shows that the proposed backbone consistently outperforms other backbones when paired with any coarse-to-fine strategy, validating its generalization and modularity.