[Paper Review] Few-Shot Learning via Learning the Representation, Provably

Simon S. Du, Wei Hu+3 more|arXiv (Cornell University)|Feb 21, 2020

Domain Adaptation and Few-Shot Learning33 references56 citations

TL;DR

The paper analyzes few-shot learning through representation learning, deriving provable sample-complexity guarantees that pool data from multiple source tasks to improve target-task learning, with explicit rates for low- and high-dimensional representations including nonlinear and neural cases.

ABSTRACT

This paper studies few-shot learning via representation learning, where one uses $T$ source tasks with $n_1$ data per task to learn a representation in order to reduce the sample complexity of a target task for which there is only $n_2 (\ll n_1)$ data. Specifically, we focus on the setting where there exists a good \emph{common representation} between source and target, and our goal is to understand how much of a sample size reduction is possible. First, we study the setting where this common representation is low-dimensional and provide a fast rate of $O\left(\frac{\mathcal{C}\left(Φ ight)}{n_1T} + \frac{k}{n_2} ight)$; here, $Φ$ is the representation function class, $\mathcal{C}\left(Φ ight)$ is its complexity measure, and $k$ is the dimension of the representation. When specialized to linear representation functions, this rate becomes $O\left(\frac{dk}{n_1T} + \frac{k}{n_2} ight)$ where $d (\gg k)$ is the ambient input dimension, which is a substantial improvement over the rate without using representation learning, i.e. over the rate of $O\left(\frac{d}{n_2} ight)$. This result bypasses the $Ω(\frac{1}{T})$ barrier under the i.i.d. task assumption, and can capture the desired property that all $n_1T$ samples from source tasks can be \emph{pooled} together for representation learning. Next, we consider the setting where the common representation may be high-dimensional but is capacity-constrained (say in norm); here, we again demonstrate the advantage of representation learning in both high-dimensional linear regression and neural network learning. Our results demonstrate representation learning can fully utilize all $n_1T$ samples from source tasks.

Motivation & Objective

• Motivate representation learning for few-shot learning to reduce target task sample complexity.
• Characterize how a common representation between source and target enables improved generalization bounds.
• Provide theoretical rates showing when and how source data can be fully exploited to aid the target task.
• Extend results from linear to nonlinear and high-dimensional settings, including neural networks.

Proposed method

• Formulate a joint optimization to learn a shared representation and task-specific predictors using source-task data: min over phi in Phi and W of (1/2n1T) sum_t ||y_t - X_t phi(X_t) w_t||^2.
• Show that the learned representation hat{phi} is then used with a target-task linear predictor: min_w 1/(2n2) ||y_{T+1} - hat{phi}(X_{T+1}) w||^2.
• Derive risk bounds for the target task that separate source-average representation error and target-specific estimation error: ER <= ~O(C(Phi)/(n1 T) + k/n2) in low-dimensional linear settings, and generalized forms for nonlinear Phi.
• Extend to high-dimensional linear representations with covariance-dominance and diversity assumptions, yielding rates involving spectral quantities of Sigma and trace terms.
• Provide neural-network capable extensions showing the same pooling benefit for two-layer ReLU networks under analogous conditions.

Experimental results

Research questions

• RQ1 What sample complexity reductions are possible in few-shot learning when a common representation exists between source and target tasks?
• RQ2 How does the size and structure of the representation (low- vs high-dimensional, linear vs nonlinear, neural networks) affect the target task risk when leveraging source-task data?
• RQ3 Under what distributional and diversity assumptions can all n1T source samples be pooled to improve target performance?
• RQ4 Do the theoretical gains extend from linear representations to nonlinear and overparameterized neural networks?

Key findings

• For low-dimensional linear representations, the target excess risk scales as ~O( dk/(n1 T) + k/n2 ), significantly improving over the vanilla d/n2 rate.
• When generalized to nonlinear representations, the bound becomes ~O( C(Phi)/(n1 T) + k/n2 ), still enabling pooling of all source data.
• Under high-dimensional linear representations with covariance structure, the rate improves to ~O( (R̄ sqrt{Tr(Sigma)})/sqrt{n1 T} + (R̄ sqrt{||Sigma||_2})/sqrt{n2} ).
• The results show that all n1T samples from source tasks can be fully utilized to learn the representation, bypassing the 1/sqrt(T) barrier under i.i.d. task assumptions.
• The framework extends to two-layer neural networks with ReLU activations, maintaining the same qualitative gains from representation learning.