[Paper Review] Regularized Learning for Domain Adaptation under Label Shifts

Kamyar Azizzadenesheli, Anqi Liu+2 more|arXiv (Cornell University)|Mar 22, 2019

Domain Adaptation and Few-Shot LearningComputer Science56 citations

TL;DR

Introduces Regularized Learning under Label Shifts (RLLS), a practical algorithm to correct for label distribution shifts between source and target using regularized weight estimation and Wasserstein-like generalization guarantees. It outperforms prior methods, especially in low-sample and large-shift regimes.

ABSTRACT

We propose Regularized Learning under Label shifts (RLLS), a principled and a practical domain-adaptation algorithm to correct for shifts in the label distribution between a source and a target domain. We first estimate importance weights using labeled source data and unlabeled target data, and then train a classifier on the weighted source samples. We derive a generalization bound for the classifier on the target domain which is independent of the (ambient) data dimensions, and instead only depends on the complexity of the function class. To the best of our knowledge, this is the first generalization bound for the label-shift problem where the labels in the target domain are not available. Based on this bound, we propose a regularized estimator for the small-sample regime which accounts for the uncertainty in the estimated weights. Experiments on the CIFAR-10 and MNIST datasets show that RLLS improves classification accuracy, especially in the low sample and large-shift regimes, compared to previous methods.

Motivation & Objective

• Motivate and address label shift in supervised learning where target labels are unavailable.
• Develop a practical weight-estimation procedure that remains robust in small-sample regimes.
• Provide a theoretical generalization bound for the target-domain classifier under label shift.
• Propose a regularized estimator to mitigate weight-estimation uncertainty and improve target performance.
• Empirically validate RLLS on MNIST and CIFAR-10 against prior methods.

Proposed method

• Define and use importance weights w(i)=q(i)/p(i) for label shifts with p and q as source/target label distributions.
• Propose a two-step weight estimation that regularizes the linear model q-hat - C-hat 1 = C-hat theta to obtain theta-hat and w = 1 + lambda theta-hat.
• Split source data into weight and class sets to estimate weights and train the classifier on the weighted empirical loss L_n(h; w).
• Derive a dimension-independent generalization bound for the RLLS classifier that depends on the function class complexity via the Rademacher complexity of the loss class and divergence terms.
• Introduce a regularized estimator w_hat = 1 + lambda theta_hat to control the influence of weight estimates in the small-sample regime, with lambda chosen based on target sample size and confidence in the weight estimates.
• Provide guidance for selecting lambda and beta to balance regularization and learning in different data-sample regimes.

Experimental results

Research questions

• RQ1 How can one accurately estimate label-shift weights q(y)/p(y) from labeled source and unlabeled target data when target labels are unavailable?
• RQ2 What are the generalization guarantees for a classifier trained on reweighted source data under label shift, and how can one account for weight-estimation uncertainty in small-sample regimes?
• RQ3 Does a regularized weight estimator improve predictive performance relative to unweighted or non-regularized weighting, especially under large shifts or limited target data?
• RQ4 How does the proposed RLLS method perform empirically on large-scale vision datasets under various label-shift scenarios compared to existing methods like BBSL?

Key findings

• RLLS achieves order-of-magnitude improvements in weight-estimation error and competitive accuracy gains over baselines in large-shift, low-sample settings.
• A dimension-independent generalization bound is derived for the RLLS classifier, improving excess-risk guarantees under label shift.
• Regularization of the weight estimator mitigates high-variance weight estimates when target data are scarce, enabling robust performance across regimes.
• Empirical results on MNIST and CIFAR-10 show that RLLS consistently outperforms BBSL in weight estimation and target accuracy, especially with large shifts and limited target data.
• Partially regularized weights (lambda between 0 and 1) can yield best performance in intermediate target-sample regimes, adapting to weight-uncertainty levels.
• The method provides practical guidance for choosing the split ratio beta, regularization strength lambda, and using a decentered predictor h_0 to bootstrap weight estimation.