[论文解读] Rewrite Caption Semantics: Bridging Semantic Gaps for Language-Supervised Semantic Segmentation

Yun Xing, Jian Kang+4 more|arXiv (Cornell University)|Sep 24, 2023

Multimodal Machine Learning Applications被引用 5

一句话总结

本文提出概念筛选（CoCu），一种三阶段流水线，通过基于CLIP的视觉驱动检索、文本到视觉引导排序以及聚类引导采样，扩展图像字幕中缺失的视觉概念，从而弥合语言监督语义分割中的语义鸿沟。CoCu在8个分割基准上显著提升零样本迁移性能，平均mIoU达到13.1%，大幅超越当前最先进方法。

ABSTRACT

Vision-Language Pre-training has demonstrated its remarkable zero-shot recognition ability and potential to learn generalizable visual representations from language supervision. Taking a step ahead, language-supervised semantic segmentation enables spatial localization of textual inputs by learning pixel grouping solely from image-text pairs. Nevertheless, the state-of-the-art suffers from clear semantic gaps between visual and textual modality: plenty of visual concepts appeared in images are missing in their paired captions. Such semantic misalignment circulates in pre-training, leading to inferior zero-shot performance in dense predictions due to insufficient visual concepts captured in textual representations. To close such semantic gap, we propose Concept Curation (CoCu), a pipeline that leverages CLIP to compensate for the missing semantics. For each image-text pair, we establish a concept archive that maintains potential visually-matched concepts with our proposed vision-driven expansion and text-to-vision-guided ranking. Relevant concepts can thus be identified via cluster-guided sampling and fed into pre-training, thereby bridging the gap between visual and textual semantics. Extensive experiments over a broad suite of 8 segmentation benchmarks show that CoCu achieves superb zero-shot transfer performance and greatly boosts language-supervised segmentation baseline by a large margin, suggesting the value of bridging semantic gap in pre-training data.

研究动机与目标

• 为解决网络爬取的图像-文本对中关键视觉概念缺失导致的语义鸿沟问题，该问题阻碍了语义分割的有效预训练。
• 通过从预训练数据中引入缺失的视觉概念，丰富文本表征，提升语言监督语义分割的零样本泛化能力。
• 开发一个鲁棒的流水线，在避免检索概念语义偏见的同时，确保视觉概念的全面且多样化覆盖。
• 证明概念筛选在提升零样本分割与零样本分类性能方面的有效性。

提出的方法

• 视觉驱动扩展通过使用CLIP嵌入从其他图像中检索相关概念，构建概念档案，确保视觉概念覆盖无偏。
• 文本到视觉引导排序利用CLIP的视觉-语言对齐能力，基于概念与输入图像的语义相关性对检索到的概念进行评分。
• 聚类引导采样通过利用语义聚类，选择多样且高相关性的概念，避免冗余并提升表征多样性。
• 将筛选后的内容整合到预训练数据中，增强模型将文本查询与准确像素级区域关联的能力。
• 将该流水线应用于从头开始在图像-文本对上预训练GroupViT，并在多个分割基准上进行零样本评估。
• 该方法利用现成的CLIP模型，避免重新训练视觉-语言编码器，实现高效且可扩展的概念筛选。

Figure 1: Cross-modal semantic gap is prevalent in web-crawled image-text pairs. As in (a), the caption text often captures certain salient visual concepts only in the paired image but misses many others (i.e., ‘ person ’, ‘ grass ’, and ‘ sky ’) that are also useful in image-text modeling. Leveragi

实验结果

研究问题

• RQ1 通过从其他图像中补充缺失的视觉概念，能否提升零样本语义分割性能？
• RQ2 如何有效检索并排序视觉概念，以最小化语义偏见，同时最大化与输入图像的相关性？
• RQ3 在概念采样过程中通过聚类引入语义多样性，是否能带来更好的零样本分割泛化性能？
• RQ4 概念筛选在超越分割任务的零样本迁移中，例如图像分类，能带来多大程度的性能提升？

主要发现

• CoCu在8个分割基准上实现13.1%的平均mIoU，较基线GroupViT提升4.9%。
• 消融实验证实，CoCu的每个组件——视觉驱动扩展、文本到视觉排序和聚类引导采样——均对性能提升有增量贡献。
• 当在C3、C12、Y14数据上预训练时，CoCu将零样本ImageNet-1K分类准确率提升6.2%（top-1）和6.9%（top-5），优于GroupViT。
• 仅视觉驱动扩展阶段即可使性能较基线提升2.1% mIoU，凸显其在减少语义偏见方面的作用。
• 聚类引导采样在所有采样策略中带来最大的边际增益，表明语义多样性对鲁棒预训练至关重要。
• 该方法在所有8个评估的分割基准上均一致提升性能，证明其广泛适用性和泛化能力。

Figure 2: Illustration of vision-driven expansion (above) and text-to-image-guided ranking (below) in CoCu. To compensate for missing semantics, vision-driven expansion establishes an archive of potential matched concepts through image-to-image retrieval, while text-to-vision-guided ranking scores r