[论文解读] A Curriculum-style Self-training Approach for Source-Free Semantic Segmentation

Yuxi Wang, Jian Liang+1 more|arXiv (Cornell University)|Jun 22, 2021

Domain Adaptation and Few-Shot Learning被引用 4

一句话总结

该论文提出ATP，一种新颖的课程学习风格自训练框架，用于无源数据语义分割，可在无需访问源数据的情况下实现域自适应。通过利用假设迁移实现的隐式特征对齐、正负伪标签的双向自训练，以及用于域内一致性的信息传播，ATP在GTA5→Cityscapes和跨城市驾驶基准上实现了最先进性能，优于需要访问源数据的方法。

ABSTRACT

Source-free domain adaptation has developed rapidly in recent years, where the well-trained source model is adapted to the target domain instead of the source data, offering the potential for privacy concerns and intellectual property protection. However, a number of feature alignment techniques in prior domain adaptation methods are not feasible in this challenging problem setting. Thereby, we resort to probing inherent domain-invariant feature learning and propose a curriculum-style self-training approach for source-free domain adaptive semantic segmentation. In particular, we introduce a curriculum-style entropy minimization method to explore the implicit knowledge from the source model, which fits the trained source model to the target data using certain information from easy-to-hard predictions. We then train the segmentation network by the proposed complementary curriculum-style self-training, which utilizes the negative and positive pseudo labels following the curriculum-learning manner. Although negative pseudo-labels with high uncertainty cannot be identified with the correct labels, they can definitely indicate absent classes. Moreover, we employ an information propagation scheme to further reduce the intra-domain discrepancy within the target domain, which could act as a standard post-processing method for the domain adaptation field. Furthermore, we extend the proposed method to a more challenging black-box source model scenario where only the source model's predictions are available. Extensive experiments validate that our method yields state-of-the-art performance on source-free semantic segmentation tasks for both synthetic-to-real and adverse conditions datasets. The code and corresponding trained models are released at \url{https://github.com/yxiwang/ATP}.

研究动机与目标

• 解决由于隐私或知识产权限制而无法获取源数据时，语义分割中的域偏移问题。
• 开发一种仅依赖预训练源模型进行目标域自适应的无源数据域自适应框架。
• 将框架扩展至更实用的黑盒场景，即仅可访问源模型的预测结果。
• 通过引入负伪标签和域内一致性正则化，改进目标域中的表示学习。

提出的方法

• 提出一种课程学习风格的熵最小化目标，通过源模型的预测结果隐式对齐目标特征与未见的源特征。
• 引入正负伪标签的双向自训练，通过聚焦于低置信度和困难样本，增强特征学习。
• 设计一种信息传播方案，应用伪半监督学习以减少目标域内的域内差异。
• 利用冻结的源分类器进行假设迁移，以稳定特征对齐并防止自适应过程中的分布偏移。
• 引入加权多样性损失，以鼓励类别无关的特征多样性，提升泛化能力。
• 通过仅使用源模型输出预测结果替代对源特征的访问，将框架扩展至黑盒设置。

Figure 1: Comparison between the conventional, source data-free, and black-box source-model domain adaptation for semantic segmentation. The proposed source data-free domain adaptation method only relies on the trained source model, while the conventional methods require all the labeled source data

实验结果

研究问题

• RQ1 当无法获取源数据时，通过假设迁移实现的隐式特征对齐是否能有效替代显式域对齐？
• RQ2 在自训练中引入负伪标签是否能提升无源数据域自适应中的表示学习？
• RQ3 通过伪半监督学习实现的信息传播是否能减少域内差异并提升分割准确率？
• RQ4 所提方法对超参数选择的鲁棒性如何，特别是在缺乏源数据的情况下？
• RQ5 该框架是否能泛化至仅可访问源模型预测结果的黑盒源模型场景？

主要发现

• ATP在GTA5→Cityscapes基准上实现了39.8的平均交并比（mIoU），超越所有先前的无源数据方法，并与需要源数据的方法性能相当。
• 该方法对超参数表现出强鲁棒性，特别是负伪标签阈值λ_neg，在λ_neg < 0.1时达到最优性能。
• 消融实验证实，隐式对齐、双向自训练和信息传播三个组件均对性能提升有显著贡献。
• 在仅可访问源模型预测结果的黑盒源模型场景中，该框架表现具有竞争力，验证了其实际可用性。
• 模型收敛迅速，且在不微调源分类器的情况下仍保持高性能，表明假设迁移机制具有高度稳定性。
• 该方法在常见类别上表现显著提升，但在罕见类别或域差距较大的类别（如围栏、电线杆、信号灯和火车）上仍存在挑战。

Figure 2: Illustration of the proposed align, teach, and propagate ( ATP ) framework for domain adaptive semantic segmentation without source data. (a) is the implicit feature alignment. We freeze the classifier and train the feature extractor to achieve implicit feature alignment by minimizing $\ma