[Paper Review] Learning to Segment 3D Point Clouds in 2D Image Space

Yecheng Lyu, Xinming Huang+1 more|arXiv (Cornell University)|Mar 12, 2020

3D Shape Modeling and Analysis72 references4 citations

TL;DR

This paper proposes a novel method to segment 3D point clouds by projecting them into 2D image space using topology-preserving graph-to-grid mapping via integer programming and graph drawing, enabling the use of standard 2D CNNs like U-Net. It achieves state-of-the-art performance on ShapeNet and PartNet with significant improvements over prior methods.

ABSTRACT

In contrast to the literature where local patterns in 3D point clouds are captured by customized convolutional operators, in this paper we study the problem of how to effectively and efficiently project such point clouds into a 2D image space so that traditional 2D convolutional neural networks (CNNs) such as U-Net can be applied for segmentation. To this end, we are motivated by graph drawing and reformulate it as an integer programming problem to learn the topology-preserving graph-to-grid mapping for each individual point cloud. To accelerate the computation in practice, we further propose a novel hierarchical approximate algorithm. With the help of the Delaunay triangulation for graph construction from point clouds and a multi-scale U-Net for segmentation, we manage to demonstrate the state-of-the-art performance on ShapeNet and PartNet, respectively, with significant improvement over the literature. Code is available at https://github.com/Zhang-VISLab.

Motivation & Objective

• To address the challenge of effectively and efficiently projecting 3D point clouds into 2D image space so that conventional 2D CNNs can be used for semantic segmentation.
• To preserve the topological structure of 3D point clouds during projection to avoid loss of local geometric patterns.
• To accelerate the computationally expensive graph drawing process by introducing a hierarchical approximation algorithm.
• To demonstrate that standard 2D CNNs, when applied to properly projected 3D point cloud images, can outperform specialized 3D deep learning models.
• To achieve state-of-the-art performance on benchmark datasets like ShapeNet and PartNet using a unified pipeline.

Proposed method

• Constructs graphs from 3D point clouds using Delaunay triangulation to encode local geometric relationships.
• Reformulates graph drawing as an integer programming problem to learn a topology-preserving mapping from 3D graphs to 2D grids.
• Proposes a novel hierarchical approximate algorithm with complexity O(n^(L+1)/L) to reduce computation time by ~97% compared to exact methods.
• Generates (x,y,z)-images by projecting 3D points onto the 2D grid using the learned mapping, with zeros for empty cells.
• Applies a multi-scale U-Net to the (x,y,z)-images for semantic segmentation, leveraging local receptive fields of 2D CNNs.
• Uses the Kamada-Kawai algorithm as a foundation for energy-based graph layout optimization, adapted to discrete 2D grids via integer programming.

Experimental results

Research questions

• RQ1 Can conventional 2D CNNs achieve state-of-the-art performance on 3D point cloud segmentation when given an appropriate 2D image representation?
• RQ2 How can 3D point cloud structure be preserved during projection into 2D image space to maintain local geometric patterns?
• RQ3 What is an efficient and scalable method to compute topology-preserving graph-to-grid mappings for large 3D point clouds?
• RQ4 Can hierarchical approximation significantly reduce the computational cost of graph drawing without sacrificing segmentation accuracy?
• RQ5 How does the proposed method compare to existing 3D-specific deep learning models on standard benchmarks?

Key findings

• The proposed method achieves state-of-the-art performance on ShapeNet, outperforming all prior methods with a mean intersection-over-union (mIoU) of 63.0%.
• On PartNet, the method improves the state-of-the-art by 6.8%, 9.1%, and 5.9% on three different segmentation levels, with an average improvement of 3.2%.
• The method achieves 88.6% class-mIoU and 65.2% instance-mIoU on ShapeNet, significantly outperforming previous SOTA results.
• The hierarchical approximation reduces running time by approximately 97% compared to the exact integer programming solution.
• The method achieves the best performance in 31 out of 50 categories across all levels in PartNet, demonstrating strong robustness and generalization.
• The multi-scale U-Net architecture effectively captures local context in the projected (x,y,z)-images, contributing to high segmentation accuracy.