[Paper Review] On the Theory of Transfer Learning: The Importance of Task Diversity

Nilesh Tripuraneni, Michael I. Jordan+1 more|arXiv (Cornell University)|Jun 20, 2020

Domain Adaptation and Few-Shot Learning34 references59 citations

TL;DR

This paper provides statistical guarantees for transfer learning via a shared representation, introducing a general notion of task diversity and a Gaussian complexity chain rule, and derives end-to-end transfer guarantees across multiple tasks.

ABSTRACT

We provide new statistical guarantees for transfer learning via representation learning--when transfer is achieved by learning a feature representation shared across different tasks. This enables learning on new tasks using far less data than is required to learn them in isolation. Formally, we consider $t+1$ tasks parameterized by functions of the form $f_j \circ h$ in a general function class $\mathcal{F} \circ \mathcal{H}$, where each $f_j$ is a task-specific function in $\mathcal{F}$ and $h$ is the shared representation in $\mathcal{H}$. Letting $C(\cdot)$ denote the complexity measure of the function class, we show that for diverse training tasks (1) the sample complexity needed to learn the shared representation across the first $t$ training tasks scales as $C(\mathcal{H}) + t C(\mathcal{F})$, despite no explicit access to a signal from the feature representation and (2) with an accurate estimate of the representation, the sample complexity needed to learn a new task scales only with $C(\mathcal{F})$. Our results depend upon a new general notion of task diversity--applicable to models with general tasks, features, and losses--as well as a novel chain rule for Gaussian complexities. Finally, we exhibit the utility of our general framework in several models of importance in the literature.

Motivation & Objective

• Motivate transfer learning when multiple tasks share a common representation to enable learning on new tasks with less data.
• Develop a two-stage ERM framework to learn a shared representation from t tasks and transfer to a new task.
• Introduce a problem-agnostic notion of task diversity and a Gaussian complexity chain rule to derive generalization bounds.
• Provide end-to-end transfer learning guarantees and illustrate with multiple models (logistic regression, neural networks, robust regression).

Proposed method

• Model tasks as f_j ∘ h with a shared representation h in H and task-specific maps f_j in F.
• Two-stage ERM: train phase learns h and task maps via nt samples; test phase learns f_0 using m samples and the learned h.
• Define task-averaged and worst-case representation differences to quantify representation similarity and transferability.
• Develop a chain rule for Gaussian complexities to decouple the complexity of F and H in generalization bounds.
• Introduce a problem-agnostic definition of task diversity that bounds transfer performance in terms of diversity and Gaussian complexities.
• Derive end-to-end transfer learning bounds (Theorem 3) showing how the transfer risk scales with problem dimensions and task diversity.

Experimental results

Research questions

• RQ1 How many samples are needed to learn a shared representation across training tasks?
• RQ2 How does transferring a representation affect sample efficiency on a new task?
• RQ3 What role does task diversity play in enabling fast transfer to unseen tasks?
• RQ4 Can generalization guarantees be extended beyond linear/quadratic settings to broader models?
• RQ5 How can Gaussian complexity be used to decouple complexity of task-specific maps from the shared representation?

Key findings

• Excess risk on the new task scales with a term involving C(H) and tC(F) divided by nt, plus a term depending on C(F)/m.
• A new notion of task diversity (ν, ε) controls the transfer bound and can yield faster rates under diverse training tasks.
• A novel chain rule for Gaussian complexities decomposes the bound into contributions from learning H and learning F.
• End-to-end transfer learning guarantees show improved rates over learning the new task in isolation when n and t are large and m is small.
• Applications include multitask logistic regression, multitask deep neural network regression, and robust regression for single-index models.
• In favorable settings, transfer learning achieves rate advantages scaling with the representation dimensionality r rather than the ambient dimension d (when r << d).