[Paper Review] CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding

Mohamed Afham, Isuru Dissanayake+4 more|arXiv (Cornell University)|Mar 1, 2022

3D Surveying and Cultural Heritage21 citations

TL;DR

CrossPoint learns transferable 3D point cloud representations by enforcing 3D–2D correspondence through self-supervised cross-modal contrastive learning, improving downstream 3D classification and segmentation without labels.

ABSTRACT

Manual annotation of large-scale point cloud dataset for varying tasks such as 3D object classification, segmentation and detection is often laborious owing to the irregular structure of point clouds. Self-supervised learning, which operates without any human labeling, is a promising approach to address this issue. We observe in the real world that humans are capable of mapping the visual concepts learnt from 2D images to understand the 3D world. Encouraged by this insight, we propose CrossPoint, a simple cross-modal contrastive learning approach to learn transferable 3D point cloud representations. It enables a 3D-2D correspondence of objects by maximizing agreement between point clouds and the corresponding rendered 2D image in the invariant space, while encouraging invariance to transformations in the point cloud modality. Our joint training objective combines the feature correspondences within and across modalities, thus ensembles a rich learning signal from both 3D point cloud and 2D image modalities in a self-supervised fashion. Experimental results show that our approach outperforms the previous unsupervised learning methods on a diverse range of downstream tasks including 3D object classification and segmentation. Further, the ablation studies validate the potency of our approach for a better point cloud understanding. Code and pretrained models are available at http://github.com/MohamedAfham/CrossPoint.

Motivation & Objective

• Address the need for label-free 3D point cloud representations applicable to classification, segmentation, and detection.
• Leverage cross-modal 3D–2D correspondence to transfer knowledge from rendered 2D views to 3D point clouds.
• Propose a joint intra-modal and cross-modal contrastive learning framework (IMID and CMID) without memory banks.
• Show generalization across synthetic and real-world datasets and backbone architectures.

Proposed method

• Two-branch architecture with a point cloud backbone (PointNet or DGCNN) and an image backbone (ResNet) plus projection heads.
• Intra-Modal Instance Discrimination (IMID): enforce invariance to random 3D point cloud transformations using NT-Xent contrastive loss on augmented views.
• Cross-Modal Instance Discrimination (CMID): align the point-cloud prototype with the rendered 2D image feature in a shared invariant space, creating hard positives for cross-modal learning.
• Overall loss L = L_IMID + L_CMID, trained end-to-end without a memory bank, discarding the image branch after pretraining for downstream tasks.
• Pretraining uses ShapeNet with rendered images; downstream tasks use only the point-cloud backbone.
• Evaluation on 3D object classification, few-shot learning, and part segmentation across synthetic and real-world datasets.

Experimental results

Research questions

• RQ1 Does 3D–2D cross-modal alignment improve unsupervised point cloud representations for downstream tasks?
• RQ2 How does combining intra-modal and cross-modal contrastive objectives compare to using either objective alone?
• RQ3 Do learned representations transfer to both synthetic and real-world 3D datasets and multiple backbones (PointNet, DGCNN)?
• RQ4 Can the approach also provide benefits in few-shot and part segmentation scenarios, and relate to cross-modal understanding with images?

Key findings

• CrossPoint outperforms prior unsupervised methods on ModelNet40 with both PointNet and DGCNN backbones in linear evaluation.
• On ScanObjectNN, CrossPoint achieves higher linear accuracy than prior methods across PointNet and DGCNN backbones.
• The joint IMID + CMID learning objective yields consistent gains over either objective alone on ModelNet40 and ScanObjectNN (e.g., +1.2% and +0.7% in the reported backbones).
• CrossPoint pretrained backbones provide a meaningful initialization for 3D part segmentation, with mean IoU of 85.5% versus 85.3% for some prior self-supervised baselines.
• In CIFAR-FS few-shot image classification, fine-tuning a CrossPoint-initialized ResNet-50 with RFS outperforms the baseline by notable margins (e.g., 5-way 1-shot and 5-shot settings).
• The method maintains robustness across backbones and demonstrates good generalization to out-of-distribution real-world data (ScanObjectNN).