[论文解读] Exploring Open-Vocabulary Semantic Segmentation without Human Labels

Jun Chen, Deyao Zhu+7 more|arXiv (Cornell University)|Jun 1, 2023

Multimodal Machine Learning Applications被引用 4

一句话总结

ZeroSeg 提出了一种新颖的开放词汇语义分割方法，通过将预训练视觉-语言模型（如 CLIP）的知识蒸馏到特定于分割的模型中，无需任何人工标注的像素级标签。通过使用多尺度特征蒸馏和实例匹配损失，其在 PASCAL VOC 2012（40.8 mIoU）、COCO（20.4 mIoU）和 PASCAL Context（20.6 mIoU）上实现了最先进的零样本性能，尽管未在目标数据集上进行微调，仍优于监督模型和现有的零样本方法。

ABSTRACT

Semantic segmentation is a crucial task in computer vision that involves segmenting images into semantically meaningful regions at the pixel level. However, existing approaches often rely on expensive human annotations as supervision for model training, limiting their scalability to large, unlabeled datasets. To address this challenge, we present ZeroSeg, a novel method that leverages the existing pretrained vision-language (VL) model (e.g. CLIP) to train open-vocabulary zero-shot semantic segmentation models. Although acquired extensive knowledge of visual concepts, it is non-trivial to exploit knowledge from these VL models to the task of semantic segmentation, as they are usually trained at an image level. ZeroSeg overcomes this by distilling the visual concepts learned by VL models into a set of segment tokens, each summarizing a localized region of the target image. We evaluate ZeroSeg on multiple popular segmentation benchmarks, including PASCAL VOC 2012, PASCAL Context, and COCO, in a zero-shot manner (i.e., no training or adaption on target segmentation datasets). Our approach achieves state-of-the-art performance when compared to other zero-shot segmentation methods under the same training data, while also performing competitively compared to strongly supervised methods. Finally, we also demonstrated the effectiveness of ZeroSeg on open-vocabulary segmentation, through both human studies and qualitative visualizations.

研究动机与目标

• 实现无需依赖昂贵人工标注像素级标签的开放词汇语义分割。
• 将图像级预训练视觉-语言模型（如 CLIP）中的视觉概念知识迁移至像素级分割任务。
• 开发一种避免逐像素监督的训练范式，同时保持高分割精度。
• 通过减少对语言上下文的依赖，提升对罕见或复合词类别（如 'bedclothes'、'motorbike'）的泛化能力。
• 证明从通用视觉-语言模型蒸馏知识可获得与监督方法或大规模预训练方法相媲美甚至更优的性能。

提出的方法

• 将输入图像划分为多尺度区域，以捕获分层视觉特征。
• 使用预训练的 CLIP 视觉编码器为每个区域提取图像级语义特征。
• 引入可学习的实例标记（segment tokens），通过多尺度特征蒸馏损失将 CLIP 的视觉概念蒸馏到这些标记中。
• 应用实例匹配损失，将每个实例标记与语义一致的图像区域对齐，提升空间一致性。
• 采用掩码自编码器以提升训练效率和特征表示能力。
• 仅使用 ImageNet-1K 图像进行端到端训练，无需任何人工标注的分割标签。

Figure 1: ZeroSeg overview. ZeroSeg is a zero-shot open-vocabulary method for semantic segmentation. The approach begins by dividing the input image into a set of multi-scale views. Each view is then individually processed by a pretrained CLIP visual encoder model to extract visual concepts. These v

实验结果

研究问题

• RQ1 能否在无逐像素监督的情况下，有效利用 CLIP 等视觉-语言模型实现零样本语义分割？
• RQ2 如何将图像级学习到的视觉概念迁移至局部化、像素级的分割任务？
• RQ3 哪些损失函数在对齐学习到的实例标记与语义有意义的图像区域方面最为有效？
• RQ4 一种完全无人工标签训练的模型是否能泛化至开放词汇设置，包括罕见或复合词类别？
• RQ5 纯蒸馏方法在零样本分割中的性能与监督方法或大规模预训练方法相比如何？

主要发现

• ZeroSeg 在 PASCAL VOC 2012 的零样本设置下实现了 40.8 的 mIoU，优于现有零样本方法。
• 其在 PASCAL Context 上达到 20.6 mIoU，在 COCO 上达到 20.4 mIoU，展现出在多样化数据集上的强大泛化能力。
• 消融实验表明，多尺度蒸馏和实例匹配损失均至关重要，两者结合可实现最精确的边界预测。
• ZeroSeg 在子词和复合词类别（如 'bedclothes'）上显著优于 GroupViT（如 'bedclothes' 上提升 18.07%），表明其对语言上下文的依赖更小。
• 尽管仅在 130 万张 ImageNet-1K 图像上训练且无任何人工标签，ZeroSeg 的性能仍可匹配或超越在 2000 万至 2600 万图像-文本对上预训练或经人工标注微调的模型。
• 定性结果与人类评估均表明，ZeroSeg 生成的分割掩码比以往的零样本方法更准确且空间上更一致。

Figure 2: Training ZeroSeg model. ZeroSeg architecture consists of a ViT encoder and two heads including a decoder head and a segmentation head. The outputs from the decoder head is used to reconstruct the masked input image during training ( i.e . , masked autoencoding [ 21 ] ), while the outputs f