[Paper Review] Unsupervised Domain Adaptation of Black-Box Source Models

Haojian Zhang, Yabin Zhang+2 more|arXiv (Cornell University)|Jan 8, 2021

Domain Adaptation and Few-Shot LearningComputer Science58 references37 citations

TL;DR

The paper introduces IterLNL, a framework for black-box unsupervised domain adaptation (B2UDA) that uses predictions from a black-box source model to iteratively label target data with noise-handling and category-wise sampling, achieving competitive results without access to source data or the source model itself.

ABSTRACT

Unsupervised domain adaptation (UDA) aims to learn models for a target domain of unlabeled data by transferring knowledge from a labeled source domain. In the traditional UDA setting, labeled source data are assumed to be available for adaptation. Due to increasing concerns for data privacy, source-free UDA is highly appreciated as a new UDA setting, where only a trained source model is assumed to be available, while labeled source data remain private. However, trained source models may also be unavailable in practice since source models may have commercial values and exposing source models brings risks to the source domain, e.g., problems of model misuse and white-box attacks. In this work, we study a subtly different setting, named Black-Box Unsupervised Domain Adaptation (B$^2$UDA), where only the application programming interface of source model is accessible to the target domain; in other words, the source model itself is kept as a black-box one. To tackle B$^2$UDA, we propose a simple yet effective method, termed Iterative Learning with Noisy Labels (IterLNL). With black-box models as tools of noisy labeling, IterLNL conducts noisy labeling and learning with noisy labels (LNL), iteratively. To facilitate the implementation of LNL in B$^2$UDA, we estimate the noise rate from model predictions of unlabeled target data and propose category-wise sampling to tackle the unbalanced label noise among categories. Experiments on benchmark datasets show the efficacy of IterLNL. Given neither source data nor source models, IterLNL performs comparably with traditional UDA methods that make full use of labeled source data.

Motivation & Objective

• Address privacy-preserving UDA where only a black-box source model API is accessible.
• Develop a robust learning-with-noisy-labels strategy tailored to unbalanced label noise from black-box predictions.
• Estimate noise rates without labeled target data and adaptively select training samples per category.
• Improve target-domain performance to approach traditional UDA and white-box UDA baselines without source data.

Proposed method

• Obtain target predictions from the black-box source model to generate noisy labels for unlabeled target data.
• Estimate the noise rate from target predictions using a rescaled probability-based metric to inform training (eq. 6–10).
• Apply category-wise sampling to select the smallest-loss samples per category for training (with per-category buffers).
• Use learning with noisy labels (LNL) to update a target model, guided by a curriculum R(n) that declines over iterations (eq. 6).
• Iteratively update the target model and re-use it as the new black-box source for subsequent labeling rounds (IterLNL loop).

Experimental results

Research questions

• RQ1 Can B2UDA achieve competitive performance when only a black-box source-model API is available without source data?
• RQ2 How can one effectively handle unbalanced label noise induced by domain shift in B2UDA?
• RQ3 What mechanisms (noise-rate estimation, category-wise sampling, iteration) drive robust learning in IterLNL?
• RQ4 How does IterLNL compare to existing B2UDA, source-free UDA, and standard UDA methods on benchmark datasets?

Key findings

• IterLNL significantly improves over existing B2UDA methods on benchmark datasets.
• IterLNL achieves results comparable to white-box UDA methods and traditional UDA despite no access to source data or source model internals.
• Category-wise sampling and iterative learning are crucial for handling highly unbalanced label noise and for avoiding category-specific failures.
• Ablation studies show the importance of noise-rate estimation, the rescale curve, and per-category sampling for robust performance.
• On VisDA-2017, Office31, and Digits tasks, IterLNL outperforms the baseline source-model and several B2UDA variants by substantial margins.