[Paper Review] LEEP: A New Measure to Evaluate Transferability of Learned Representations

Cuong V. Nguyen, Tal Hassner+2 more|arXiv (Cornell University)|Feb 27, 2020

Domain Adaptation and Few-Shot Learning59 references65 citations

TL;DR

LEEP estimates transferability of a source model to a target task with a single forward pass and no target training, predicting transfer performance and convergence. It outperforms NCE and H scores in correlating with actual transfer accuracy.

ABSTRACT

We introduce a new measure to evaluate the transferability of representations learned by classifiers. Our measure, the Log Expected Empirical Prediction (LEEP), is simple and easy to compute: when given a classifier trained on a source data set, it only requires running the target data set through this classifier once. We analyze the properties of LEEP theoretically and demonstrate its effectiveness empirically. Our analysis shows that LEEP can predict the performance and convergence speed of both transfer and meta-transfer learning methods, even for small or imbalanced data. Moreover, LEEP outperforms recently proposed transferability measures such as negative conditional entropy and H scores. Notably, when transferring from ImageNet to CIFAR100, LEEP can achieve up to 30% improvement compared to the best competing method in terms of the correlations with actual transfer accuracy.

Motivation & Objective

• Motivate the need for reliable, inexpensive transferability estimation for deep representations.
• Introduce LEEP as a measure that requires only a single forward pass on the target data.
• Provide theoretical properties linking LEEP to optimal transfer performance and to NCE.
• Empirically validate LEEP across various transfer and meta-transfer scenarios, including small/imbalanced data.
• Show practical usefulness for source model selection and convergence prediction.

Proposed method

• Define LEEP as the average log-likelihood of the Expected Empirical Predictor (EEP) built from a source model and the target data.
• Compute dummy source-label distributions by running the target data through the source model.
• Estimate empirical conditional P(y|z) from observed target labels and dummy source labels.
• Compute T(θ, D) = (1/n) sum_i log(sum_z P(y_i|z) θ(x_i)_z) as the transferability score.
• Establish that LEEP is a lower bound on the optimal retrained model’s average log-likelihood (with EEP included in hypothesis space).
• Relate LEEP to negative conditional entropy (NCE) and compare interpretability and computation to prior measures.

Experimental results

Research questions

• RQ1 Can LEEP predict the performance of common transfer learning methods (head retraining and fine-tuning) withoutTARGET task training?
• RQ2 Does LEEP correlate with meta-transfer learning performance (e.g., CNAPs) and with convergence speed in fine-tuning?
• RQ3 How does LEEP compare to NCE and H scores across varied data regimes (small, imbalanced, noisy) and source/target pairs?
• RQ4 Is LEEP useful for selecting which source models to deploy for a given target task?

Key findings

• LEEP scores correlate strongly with transfer accuracies across tasks and transfer methods, with Pearson correlations > 0.94 and p < 0.001 in many settings.
• LEEP remains predictive in small and imbalanced target data regimes and even with noisy labels.
• LEEP predicts faster convergence of fine-tuning and can indicate when transferred models surpass reference models trained from scratch.
• LEEP outperforms NCE and H scores in most comparisons, with up to 30% improvement in correlation with actual transfer accuracy.
• LEEP also provides a viable measure for meta-transfer learning (CNAPs) performance, showing significant correlation (0.591, p < 0.001).
• In source model selection, LEEP-based estimates better align with transfer performance than NCE or H scores in many settings.