[Paper Review] A Curriculum-style Self-training Approach for Source-Free Semantic Segmentation

Yuxi Wang, Jian Liang+1 more|arXiv (Cornell University)|Jun 22, 2021

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes ATP, a novel curriculum-style self-training framework for source-free semantic segmentation that enables domain adaptation without access to source data. By leveraging implicit feature alignment via hypothesis transfer, bidirectional self-training with positive and negative pseudo-labels, and information propagation for intra-domain consistency, ATP achieves state-of-the-art performance on GTA5→Cityscapes and cross-city driving benchmarks, outperforming methods requiring source data access.

ABSTRACT

Source-free domain adaptation has developed rapidly in recent years, where the well-trained source model is adapted to the target domain instead of the source data, offering the potential for privacy concerns and intellectual property protection. However, a number of feature alignment techniques in prior domain adaptation methods are not feasible in this challenging problem setting. Thereby, we resort to probing inherent domain-invariant feature learning and propose a curriculum-style self-training approach for source-free domain adaptive semantic segmentation. In particular, we introduce a curriculum-style entropy minimization method to explore the implicit knowledge from the source model, which fits the trained source model to the target data using certain information from easy-to-hard predictions. We then train the segmentation network by the proposed complementary curriculum-style self-training, which utilizes the negative and positive pseudo labels following the curriculum-learning manner. Although negative pseudo-labels with high uncertainty cannot be identified with the correct labels, they can definitely indicate absent classes. Moreover, we employ an information propagation scheme to further reduce the intra-domain discrepancy within the target domain, which could act as a standard post-processing method for the domain adaptation field. Furthermore, we extend the proposed method to a more challenging black-box source model scenario where only the source model's predictions are available. Extensive experiments validate that our method yields state-of-the-art performance on source-free semantic segmentation tasks for both synthetic-to-real and adverse conditions datasets. The code and corresponding trained models are released at \url{https://github.com/yxiwang/ATP}.

Motivation & Objective

• Address the challenge of domain shift in semantic segmentation when source data is unavailable due to privacy or intellectual property constraints.
• Develop a source data-free domain adaptation framework that relies only on a pre-trained source model for adaptation to a new target domain.
• Extend the framework to the more practical black-box scenario where only the source model's predictions are accessible.
• Improve representation learning in the target domain by introducing negative pseudo-labels and intra-domain consistency regularization.

Proposed method

• Propose a curriculum-style entropy minimization objective to implicitly align target features with unseen source features using the source model’s predictions.
• Introduce bidirectional self-training with both positive and negative pseudo-labels to enhance feature learning by focusing on low-confidence and hard samples.
• Design an information propagation scheme that applies pseudo-semi-supervised learning to reduce intra-domain discrepancy within the target domain.
• Utilize hypothesis transfer with a frozen source classifier to stabilize feature alignment and prevent distribution shift during adaptation.
• Introduce a weighted diversity loss to encourage class-agnostic feature diversity and improve generalization.
• Extend the framework to the black-box setting by replacing access to source features with only the source model’s output predictions.

Figure 1: Comparison between the conventional, source data-free, and black-box source-model domain adaptation for semantic segmentation. The proposed source data-free domain adaptation method only relies on the trained source model, while the conventional methods require all the labeled source data

Experimental results

Research questions

• RQ1 Can implicit feature alignment via hypothesis transfer effectively replace explicit domain alignment when source data is unavailable?
• RQ2 Does incorporating negative pseudo-labels in self-training improve representation learning in source-free domain adaptation?
• RQ3 Can information propagation via pseudo-semi-supervised learning reduce intra-domain discrepancy and improve segmentation accuracy?
• RQ4 How robust is the proposed method to hyper-parameter choices, particularly in the absence of source data?
• RQ5 Can the framework generalize to the black-box source model scenario where only predictions are available?

Key findings

• ATP achieves a mean Intersection over Union (mIoU) of 39.8 on the GTA5→Cityscapes benchmark, surpassing all prior source data-free methods and matching performance of methods that require source data.
• The method demonstrates strong robustness to hyper-parameters, particularly the negative pseudo-labeling threshold λ_neg, with optimal performance at λ_neg < 0.1.
• Ablation studies confirm that each component—implicit alignment, bidirectional self-training, and information propagation—contributes significantly to performance gains.
• The framework performs competitively in the black-box source model scenario, where only predictions from the source model are available, validating its practicality.
• The model converges quickly and maintains high performance even when the source classifier is not fine-tuned, indicating the stability of the hypothesis transfer mechanism.
• The method shows particular improvement on common classes but still struggles with rare or large-domain-gap categories like fence, pole, signal, and train.

Figure 2: Illustration of the proposed align, teach, and propagate ( ATP ) framework for domain adaptive semantic segmentation without source data. (a) is the implicit feature alignment. We freeze the classifier and train the feature extractor to achieve implicit feature alignment by minimizing $\ma