[Paper Review] PeP: a Point enhanced Painting method for unified point cloud tasks

Zichao Dong, Hang Ji+4 more|arXiv (Cornell University)|Oct 11, 2023

3D Surveying and Cultural Heritage4 citations

TL;DR

PeP proposes a unified, model-agnostic framework for point cloud recognition by enhancing LiDAR features through a refined point painting method and a language model-based point encoder. It achieves state-of-the-art performance on both LiDAR semantic segmentation (nuScenes mIoU: 79.8) and multi-modal 3D object detection (KITTI 3D AP: 95.57/90.98/91.01) by leveraging open-vocabulary segmentation for semantic/instance labels and self-attention to fuse point attributes into contextual embeddings.

ABSTRACT

Point encoder is of vital importance for point cloud recognition. As the very beginning step of whole model pipeline, adding features from diverse sources and providing stronger feature encoding mechanism would provide better input for downstream modules. In our work, we proposed a novel PeP module to tackle above issue. PeP contains two main parts, a refined point painting method and a LM-based point encoder. Experiments results on the nuScenes and KITTI datasets validate the superior performance of our PeP. The advantages leads to strong performance on both semantic segmentation and object detection, in both lidar and multi-modal settings. Notably, our PeP module is model agnostic and plug-and-play. Our code will be publicly available soon.

Motivation & Objective

• To improve point cloud recognition by enhancing feature representation through multi-source semantic priors.
• To address the limitations of traditional point painting that rely on paired images or fixed label sources.
• To unify point cloud tasks (segmentation, detection) under a single, plug-and-play framework.
• To explore the integration of language modeling principles into point cloud feature learning for improved contextual representation.
• To develop a data-efficient, scalable method that is agnostic to downstream architectures.

Proposed method

• A refined point painting method injects semantic and instance labels from open-vocabulary segmentation models (e.g., ODISE) into LiDAR points as additional features.
• A tokenizer converts raw point features (e.g., x, y, z, intensity) into learnable embeddings of dimension d=4.
• A single-layer self-attention mechanism enables inter-attribute communication within each point, forming contextualized point embeddings.
• The LM-based point encoder operates independently and can be plugged into any backbone network without architectural changes.
• Painting is applied in two stages: first for semantic labeling, then for feature injection with both semantic and instance IDs.
• The framework supports both LiDAR-only and multi-modal settings by using image-based segmentation for label generation.

Experimental results

Research questions

• RQ1 Can a language model-inspired encoding scheme improve point cloud feature representation beyond standard geometric features?
• RQ2 How does incorporating semantic and instance labels from open-vocabulary segmentation models affect point cloud recognition performance?
• RQ3 Can a unified, plug-and-play module enhance both 3D object detection and semantic segmentation tasks simultaneously?
• RQ4 What is the impact of multi-source semantic priors (image or self-supervised) on feature quality and model convergence?
• RQ5 How does the proposed method scale across different datasets and modalities, especially under data-scarce conditions?

Key findings

• PeP achieves state-of-the-art performance on the nuScenes validation set, with mIoU of 79.8 compared to the baseline SphereFormer's 78.4.
• On the KITTI 3D detection benchmark, PeP improves 3D AP to 95.57 (easy), 90.98 (medium), and 91.01 (hard), outperforming the baseline VirConv.
• The method demonstrates strong generalization across tasks, improving both semantic segmentation and 3D object detection in both LiDAR-only and multi-modal settings.
• The LM-based point encoder alone boosts performance, but gains are significantly larger when combined with the refined painting module.
• The framework is model-agnostic and plug-and-play, enabling integration into existing point cloud pipelines without architectural modifications.
• Despite strong gains, convergence is more challenging on KITTI, suggesting the LM-based encoder may require more data for optimal performance.