[Paper Review] P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Prompting

Ziyi Wang, Xumin Yu+3 more|arXiv (Cornell University)|Aug 4, 2022

3D Surveying and Cultural Heritage37 citations

TL;DR

A method to leverage frozen pre-trained 2D image models for 3D point cloud tasks by converting point clouds to colorful images via geometry-preserved projection and geometry-aware coloring, enabling low-parameter tuning and strong performance.

ABSTRACT

Nowadays, pre-training big models on large-scale datasets has become a crucial topic in deep learning. The pre-trained models with high representation ability and transferability achieve a great success and dominate many downstream tasks in natural language processing and 2D vision. However, it is non-trivial to promote such a pretraining-tuning paradigm to the 3D vision, given the limited training data that are relatively inconvenient to collect. In this paper, we provide a new perspective of leveraging pre-trained 2D knowledge in 3D domain to tackle this problem, tuning pre-trained image models with the novel Point-to-Pixel prompting for point cloud analysis at a minor parameter cost. Following the principle of prompting engineering, we transform point clouds into colorful images with geometry-preserved projection and geometry-aware coloring to adapt to pre-trained image models, whose weights are kept frozen during the end-to-end optimization of point cloud analysis tasks. We conduct extensive experiments to demonstrate that cooperating with our proposed Point-to-Pixel Prompting, better pre-trained image model will lead to consistently better performance in 3D vision. Enjoying prosperous development from image pre-training field, our method attains 89.3% accuracy on the hardest setting of ScanObjectNN, surpassing conventional point cloud models with much fewer trainable parameters. Our framework also exhibits very competitive performance on ModelNet classification and ShapeNet Part Segmentation. Code is available at https://github.com/wangzy22/P2P.

Motivation & Objective

• Address data scarcity in 3D point cloud pre-training by transferring knowledge from 2D image pre-training.
• Introduce a prompting-based pipeline that converts point clouds into colorized images compatible with frozen 2D models.
• Enable end-to-end optimization with minimal trainable parameters while preserving geometric information.

Proposed method

• Encode point cloud geometry with a lightweight DGCNN to obtain per-point features.
• Apply geometry-preserved projection to map point features into an image-style layout while retaining spatial information.
• Use a geometry-aware coloring module to color the projected image based on geometry, producing a colorful image I for the pre-trained image model.
• Feed the colorful image into a frozen pre-trained image model (e.g., ViT, ConvNeXt) to extract image-level features for downstream heads.
• Train task-specific heads (classification, segmentation) with losses defined in the 2D image domain and project results back to 3D as needed.

Experimental results

Research questions

• RQ1 Can pre-trained 2D image models be effectively used for 3D point cloud analysis by converting point clouds to colorized images?
• RQ2 How does the scale of the pre-trained image model affect 3D point cloud classification performance under the P2P framework?
• RQ3 What is the impact of freezing the image model vs. fine-tuning only select parameters in achieving good 3D performance?
• RQ4 Does the geometry-preserved projection preserve sufficient 3D structure to support accurate downstream tasks like segmentation?
• RQ5 How do different pre-training strategies in 2D (supervised vs. unsupervised) transfer to 3D tasks via P2P?

Key findings

• Enlarging the scale of the same pre-trained image model improves 3D classification performance under P2P.
• P2P with HorNet-L pre-trained on ImageNet-22k achieves 89.3% on ScanObjectNN, surpassing traditional 3D pre-training baselines.
• P2P delivers competitive results on ModelNet40 classification and ShapeNetPart segmentation with far fewer trainable parameters.
• Colorful image prompting (Point-to-Pixel) is more effective than using geometric features alone or direct patch embedding on features.
• Supervised 2D pre-training generally yields better 3D transfer than unsupervised pre-training, though strong unsupervised methods (e.g., DINO, MAE) are competitive.
• Ablations show that tuning only normalization parameters of the image model provides a favorable balance between leveraging 2D knowledge and avoiding overfitting.