[论文解读] Hyperbolic Contrastive Learning for Hierarchical 3D Point Cloud Embedding

Yingjie Liu, Pengyu Zhang+4 more|arXiv (Cornell University)|Jan 4, 2025

3D Shape Modeling and Analysis被引用 3

一句话总结

论文通过超球面、层级感知对比学习来处理3D点云，结合蕴涵正则化和重建引导的跨模态对齐（文本与图像），提升下游3D任务表现。

ABSTRACT

Hyperbolic spaces allow for more efficient modeling of complex, hierarchical structures, which is particularly beneficial in tasks involving multi-modal data. Although hyperbolic geometries have been proven effective for language-image pre-training, their capabilities to unify language, image, and 3D Point Cloud modalities are under-explored. We extend the 3D Point Cloud modality in hyperbolic multi-modal contrastive pre-training. Additionally, we explore the entailment, modality gap, and alignment regularizers for learning hierarchical 3D embeddings and facilitating the transfer of knowledge from both Text and Image modalities. These regularizers enable the learning of intra-modal hierarchy within each modality and inter-modal hierarchy across text, 2D images, and 3D Point Clouds. Experimental results demonstrate that our proposed training strategy yields an outstanding 3D Point Cloud encoder, and the obtained 3D Point Cloud hierarchical embeddings significantly improve performance on various downstream tasks.

研究动机与目标

• 设计嵌入空间以尊重多模态数据（文本、图像、3D点云）的层级结构的动机。
• 将超曲线对比预训练扩展到3D点云与跨模态层级。
• 开发正则化项以在模态内和模态之间强制层级关系。
• 利用重建引导来稳定并提升3D点云嵌入学习。

提出的方法

• 采用 Lorentz（双曲面）模型进行超曲线嵌入表示文本、图像和3D点云模态。
• 使用重建引导的对比学习（受 ReCon 启发）在训练3D点云编码器时蒸馏教师的集成知识。
• 引入增强超曲线性的正则化项：跨文本–图像–点云的蕴涵损失和基于质心的层级约束。
• 通过蕴涵正则化分析模态内和模态间的层级关系，强制超曲线空间中的圆锥关系。
• 用 Gromov delta-超曲率来量化超曲线性，以评估嵌入的树状结构。
• 通过基于同化误差的权重分配自动平衡多种损失。

(a) Distribution of embedding distances between text and 3D point cloud embeddings shows the whole $\rightarrow$ part composition relation.

实验结果

研究问题

• RQ1 RQ1：所提出的超曲线、层级感知框架在分层3D点云嵌入中带来哪些优势？
• RQ2 RQ2：相较于当前最先进方法，分层3D点云嵌入对下游3D点云任务的影响如何？

主要发现

• 文本、图像和3D点云的嵌入呈现超曲线结构，且在训练过程中超曲率持续演化。
• 正则化项有效构建模态内和模态间的层级关系并维持模态间差距。
• 基于重建引导的跨模态训练提升了3D点云的下游任务，包括分割与分类基准。
• 该框架通过层级感知的损失和对齐到超曲线文本–图像嵌入，实现对3D点云的整体到部分的显式组合推理。

(b) Distribution of embedding distances between text, image, and 3D point cloud embeddings demonstrates that the inter-modal hierarchical relationship.