[论文解读] Connecting Multi-modal Contrastive Representations

Zehan Wang, Yang Zhao+9 more|arXiv (Cornell University)|May 22, 2023

Multimodal Machine Learning Applications被引用 13

一句话总结

C-MCR 通过通过一个重叠模态连接现有的跨模态对比表示，在不需要成对数据的情况下学习跨模态表示，在音视频任务和3D-语言任务上实现零样本的最先进结果。

ABSTRACT

Multi-modal Contrastive Representation learning aims to encode different modalities into a semantically aligned shared space. This paradigm shows remarkable generalization ability on numerous downstream tasks across various modalities. However, the reliance on massive high-quality data pairs limits its further development on more modalities. This paper proposes a novel training-efficient method for learning MCR without paired data called Connecting Multi-modal Contrastive Representations (C-MCR). Specifically, given two existing MCRs pre-trained on (A, B) and (B, C) modality pairs, we project them to a new space and use the data from the overlapping modality B to aligning the two MCRs in the new space. Meanwhile, since the modality pairs (A, B) and (B, C) are already aligned within each MCR, the connection learned by overlapping modality can also be transferred to non-overlapping modality pair (A, C). To unleash the potential of C-MCR, we further introduce a semantic-enhanced inter- and intra-MCR connection method. We first enhance the semantic consistency and completion of embeddings across different modalities for more robust alignment. Then we utilize the inter-MCR alignment to establish the connection, and employ the intra-MCR alignment to better maintain the connection for inputs from non-overlapping modalities. To demonstrate the effectiveness of C-MCR, we connect CLIP and CLAP via texts to derive audio-visual representations, and integrate CLIP and ULIP via images for 3D-language representations. Remarkably, without using any paired data, C-MCR for audio-visual achieves state-of-the-art performance on audio-image retrieval, audio-visual source localization, and counterfactual audio-image recognition tasks. Furthermore, C-MCR for 3D-language also attains advanced zero-shot 3D point cloud classification accuracy on ModelNet40.

研究动机与目标

• 在成对数据稀缺或不可用时，推动学习稳健的多模态表示。
• 提出一种轻量级方法，通过重叠模态连接预训练的 MCR 空间。
• 通过跨模态和 intra-MCR 策略增强语义对齐，缩小模态差距。
• 在音视频和3D-语言任务上展示该方法，展示出强大的零样本性能。

提出的方法

• 将 C-MCR 正式化为学习两个简单的投影器，将来自两个预训练 MCR 的嵌入映射到共享空间。
• 引入语义增强，包含跨模态语义一致性和模态内语义补全。
• 使用文本引导投影和两个对比损失（L_ttc 与 L_avc）建立跨-MCR 对齐。
• 通过模态内对齐来保持对非重叠模态的连接，从而缩小模态差距。
• 使用冻结的编码器和离线内存进行训练，仅优化两个投影器，联合损失为 L = L_inter + lambda L_intra。
• 将该框架应用于连接 CLIP 与 CLAP 以实现音视频任务，以及将 CLIP 与 ULIP 结合用于 3D-语言任务。

实验结果

研究问题

• RQ1 在不依赖大规模成对数据的情况下，是否可以将现有的 MCR 空间连接起来？
• RQ2 如何利用重叠模态将对齐转移到非重叠模态对？
• RQ3 语义增强和模态内对齐是否提高了所学连接的鲁棒性和迁移性？
• RQ4 C-MCR 在音视频和3D-语言任务上可以实现哪些零样本性能提升？

主要发现

• C-MCR 在音视频任务上实现了零样本性能的最新水平，且训练阶段不需要任何成对数据。
• 在音视频任务上，C-MCR 在六个数据集和三个下游任务（音视频检索、定位和反事实识别）中获得强大的零样本结果。
• 在3D-语言方面，C-MCR 获得先进的零样本 ModelNet40 分类精度。
• 该方法仅使用两个可学习的投影器且编码器冻结，从而训练高效且参数量小。
• 一个语义增强的跨-MCR 与模态内连接实现了 CLIP/CLAP 之间以及 ULIP/CLIP 通过图像实现的可迁移对齐。