[论文解读] Mix-and-Match Tuning for Self-Supervised Semantic Segmentation

Xiaohang Zhan, Ziwei Liu+3 more|arXiv (Cornell University)|Dec 2, 2017

Advanced Neural Network Applications参考文献 35被引用 6

一句话总结

本文提出混合匹配（Mix-and-Match，M&M）微调方法，一种即插即用的策略，通过利用目标数据集中稀疏且混合的图像块，构建类别相关的连通图，以实现基于三元组的度量学习，从而提升自监督语义分割性能。该方法在PASCAL VOC2012和CityScapes数据集上均超越了ImageNet监督预训练的性能，实现了当前最优结果，且无需额外标注。

ABSTRACT

Deep convolutional networks for semantic image segmentation typically require large-scale labeled data, e.g. ImageNet and MS COCO, for network pre-training. To reduce annotation efforts, self-supervised semantic segmentation is recently proposed to pre-train a network without any human-provided labels. The key of this new form of learning is to design a proxy task (e.g. image colorization), from which a discriminative loss can be formulated on unlabeled data. Many proxy tasks, however, lack the critical supervision signals that could induce discriminative representation for the target image segmentation task. Thus self-supervision's performance is still far from that of supervised pre-training. In this study, we overcome this limitation by incorporating a "mix-and-match" (M&M) tuning stage in the self-supervision pipeline. The proposed approach is readily pluggable to many self-supervision methods and does not use more annotated samples than the original process. Yet, it is capable of boosting the performance of target image segmentation task to surpass fully-supervised pre-trained counterpart. The improvement is made possible by better harnessing the limited pixel-wise annotations in the target dataset. Specifically, we first introduce the "mix" stage, which sparsely samples and mixes patches from the target set to reflect rich and diverse local patch statistics of target images. A "match" stage then forms a class-wise connected graph, which can be used to derive a strong triplet-based discriminative loss for fine-tuning the network. Our paradigm follows the standard practice in existing self-supervised studies and no extra data or label is required. With the proposed M&M approach, for the first time, a self-supervision method can achieve comparable or even better performance compared to its ImageNet pre-trained counterpart on both PASCAL VOC2012 dataset and CityScapes dataset.

研究动机与目标

• 解决自监督与全监督预训练在语义分割中性能差距的问题。
• 通过更有效地利用目标数据集中的有限像素级标注，提升自监督模型的判别性表征学习能力。
• 开发一种即插即用的微调阶段，增强自监督特征表示，且无需额外标签或数据。
• 证明当结合M&M微调时，自监督预训练可超越ImageNet预训练的性能。

提出的方法

• ‘混合’阶段从目标数据集中随机采样并组合局部图像块，形成多样化的图像块统计特性集合。
• ‘匹配’阶段基于混合后的图像块构建类别相关的连通图，以发现用于度量学习的困难三元组。
• 在图上计算基于三元组的判别性损失，以微调网络，提升不同类别间特征的分离度。
• 该方法在自监督代理预训练之后、标准微调之前应用，无需额外标注。
• 该方法兼容多种自监督代理任务（如颜色化、拼图）和主干网络（如AlexNet、VGG-16）。
• 图的大小可调节，允许在性能与训练效率之间进行权衡。

实验结果

研究问题

• RQ1 能否在不引入额外标注的前提下，使自监督预训练在语义分割中达到与ImageNet监督预训练相当的性能？
• RQ2 如何更好地利用目标数据集中有限的像素级标注，以提升自监督特征学习效果？
• RQ3 类似混合匹配这样的多阶段优化策略是否能超越标准微调，显著提升特征的可分性？
• RQ4 能否通过基于目标数据特定的图结构度量学习，有效优化如颜色化等代理任务，从而缩小与监督预训练的性能差距？
• RQ5 图的大小与图像块混合策略对所学表征质量有何影响？

主要发现

• 在PASCAL VOC2012上，M&M微调将mIoU从颜色化预训练的38.4%提升至42.8%（使用VGG-16），超越了随机初始化的48.0% mIoU，接近ImageNet预训练的64.2% mIoU。
• 在CityScapes上，M&M微调将mIoU从颜色化预训练的57.5%提升至66.4%，接近ImageNet预训练的67.9% mIoU。
• 对于随机初始化的网络，M&M微调在CityScapes上将mIoU从42.5%提升至49.1%，表明即使从弱初始化开始也能实现显著性能提升。
• t-SNE可视化结果证实，M&M微调显著改善了类别内特征的聚类效果，使语义上不同的类别间分离度更高。
• 更大的图结构可提升性能，但会增加训练时间，表明准确率与效率之间存在可调和的权衡。
• 该方法在多个基准和主干网络上均实现了当前最优结果，证明其在不同代理任务和数据集上的广泛有效性。