[Paper Review] LogME: Practical Assessment of Pre-trained Models for Transfer Learning

Kaichao You, Yong Liu+2 more|arXiv (Cornell University)|Feb 22, 2021

Topic Modeling72 references39 citations

TL;DR

LogME introduces a fast, general method to assess pre-trained models for transfer learning by estimating the maximum evidence (LogME) of labels given features, enabling model selection without fine-tuning across supervised, unsupervised, classification, regression, and language tasks.

ABSTRACT

This paper studies task adaptive pre-trained model selection, an underexplored problem of assessing pre-trained models for the target task and select best ones from the model zoo \emph{without fine-tuning}. A few pilot works addressed the problem in transferring supervised pre-trained models to classification tasks, but they cannot handle emerging unsupervised pre-trained models or regression tasks. In pursuit of a practical assessment method, we propose to estimate the maximum value of label evidence given features extracted by pre-trained models. Unlike the maximum likelihood, the maximum evidence is \emph{immune to over-fitting}, while its expensive computation can be dramatically reduced by our carefully designed algorithm. The Logarithm of Maximum Evidence (LogME) can be used to assess pre-trained models for transfer learning: a pre-trained model with a high LogME value is likely to have good transfer performance. LogME is \emph{fast, accurate, and general}, characterizing itself as the first practical method for assessing pre-trained models. Compared with brute-force fine-tuning, LogME brings at most $3000 imes$ speedup in wall-clock time and requires only $1\%$ memory footprint. It outperforms prior methods by a large margin in their setting and is applicable to new settings. It is general enough for diverse pre-trained models (supervised pre-trained and unsupervised pre-trained), downstream tasks (classification and regression), and modalities (vision and language). Code is available at this repository: \href{https://github.com/thuml/LogME}{https://github.com/thuml/LogME}.

Motivation & Objective

• Motivate task adaptive pre-trained model selection to choose the best starting model for a target task without fine-tuning.
• Propose a probabilistic, gradient-free score (LogME) based on the maximum evidence to assess feature-label compatibility.
• Demonstrate LogME's applicability to supervised and unsupervised pre-trained models, as well as classification and regression tasks across modalities.
• Show that LogME provides large speedups over brute-force fine-tuning while correlating strongly with actual transfer performance.

Proposed method

• Represent a pre-trained model as a fixed feature extractor and model the target with a linear predictor on top of extracted features.
• Compute the evidence p(y|F) by marginalizing over weights with Gaussian priors, leading to a closed-form log-evidence L(alpha,beta).
• Use a fixed-point alternating optimization to maximize L with respect to alpha and beta, yielding gamma and an efficient update for m and A.
• Define LogME as the average max evidence per target dimension, i.e., LogME = (1/K) sum_k L_k / n after solving for each output dimension k.
• Extend LogME to multivariate targets by applying it per dimension and averaging, and treat classification by one-hot encoding to fit a regression framework.
• Provide computational optimizations to avoid matrix inversions by leveraging the SVD of F^T F to achieve matrix-vector updates.

Experimental results

Research questions

• RQ1 Can LogME reliably rank pre-trained models for transfer learning without fine-tuning across diverse tasks and modalities?
• RQ2 Is LogME fast and memory-efficient enough to outperform brute-force fine-tuning while maintaining strong correlation with actual transfer performance?
• RQ3 Does LogME extend to supervised, unsupervised, classification, regression, and language-model scenarios?
• RQ4 How does LogME compare to prior methods like LEEP and NCE in terms of applicability and predictive accuracy?

Key findings

• LogME consistently shows higher correlation with fine-tuned transfer performance than prior methods on multiple datasets and tasks.
• LogME achieves up to about 3000× wall-clock speedup and only 1% memory footprint versus brute-force fine-tuning for model selection.
• LogME is applicable to supervised and unsupervised pre-trained models, and to classification, regression, and language-model tasks across vision and language domains.
• In regression (dSprites) LogME correlates with lower MSE after fine-tuning with 0.84 tau, implying 92% probability that a higher-LogME model yields better performance.
• For GLUE tasks with language models, LogME can perfectly estimate the ranking for 3 tasks (tau_w = 1) using transferred results from external tuning efforts.
• Empirical results show LogME provides practical, task-adaptive model selection that aligns with ground-truth transfer performance without the need for hyper-parameter tuning on the target task.