[Paper Review] On the Value of Target Data in Transfer Learning

Steve Hanneke, Samory Kpotufe|arXiv (Cornell University)|Feb 12, 2020

Domain Adaptation and Few-Shot Learning16 citations

TL;DR

This paper establishes minimax rates for transfer learning in terms of both source and target sample sizes, introducing novel discrepancy measures called transfer exponents ($\rho$ and $\gamma$) to quantify the value of labeled and unlabeled target data. It shows that optimal performance is achieved by ignoring the less informative of the source or target dataset, and proposes a data-driven method that achieves near-minimax performance without estimating distributional parameters or $\gamma$, enabling cost-minimal sampling decisions.

ABSTRACT

We aim to understand the value of additional labeled or unlabeled target data in transfer learning, for any given amount of source data; this is motivated by practical questions around minimizing sampling costs, whereby, target data is usually harder or costlier to acquire than source data, but can yield better accuracy. To this aim, we establish the first minimax-rates in terms of both source and target sample sizes, and show that performance limits are captured by new notions of discrepancy between source and target, which we refer to as transfer exponents.

Motivation & Objective

• To understand the value of labeled and unlabeled target data in transfer learning, especially under sampling cost constraints.
• To establish minimax rates for transfer learning that depend on both source ($n_P$) and target ($n_Q$) sample sizes.
• To develop practical decision-making strategies—such as sampling cost minimization and source selection—without requiring knowledge of distributional parameters or discrepancy measures.
• To show that optimal performance can be achieved by a simple classifier that ignores one dataset, provided the transfer exponent is known.

Proposed method

• Introduces two new discrepancy measures: the excess risk transfer exponent $\rho$ and the marginal transfer exponent $\gamma$, which quantify distributional differences between source and target.
• Derives minimax rates for general bounded-VC classes in terms of $n_P$, $n_Q$, and the transfer exponents $\rho$ and $\gamma$.
• Proposes a hypothesis-transfer method that optimizes target error under a constraint of low source error, without requiring knowledge of $\rho$ or $\gamma$.
• Uses unlabeled target data ($U_Q$) to estimate a reweighting scheme that enables near-optimal performance with low sampling complexity.
• Applies a robustness condition (RCS) and Bernstein class assumptions to bound excess risk and ensure convergence under minimal assumptions.
• Employs a data-driven selection rule that minimizes an upper bound on excess risk, using only unlabeled data and labeled source data.

Experimental results

Research questions

• RQ1 What is the optimal rate of convergence for transfer learning when both source and target labeled data are available?
• RQ2 How does the performance of a classifier combining source and target data compare to using only one dataset?
• RQ3 Can practical decisions in transfer learning—such as sampling cost minimization or source selection—be made without estimating distributional parameters or discrepancy measures?
• RQ4 What role do unlabeled target samples play in achieving minimax-optimal performance in transfer learning?

Key findings

• The minimax-optimal rate for target error is achieved—up to constants—by using only the more informative of the source or target dataset, even when both are available.
• The optimal rate in terms of $n_P$ and $n_Q$ is captured by the transfer exponent $\rho$, which measures discrepancy in excess risk between source and target.
• A data-driven procedure that minimizes an upper bound on excess risk achieves the minimax rate without estimating $\rho$ or $\gamma$, enabling practical deployment.
• The marginal transfer exponent $\gamma$, defined on input marginals $P_X$ and $Q_X$, enables optimal decisions in scenarios involving unlabeled data and cost minimization.
• Unlabeled target data of size $|U_Q| \geq n_P^{\frac{\beta_f}{(2 - \beta_f)\gamma_f}}$ suffices to achieve the minimax rate, with low sampling complexity.
• The method remains effective even when label noise or distributional shifts are unknown, and does not require estimation of $\gamma$.