[Paper Review] Repurposing Pretrained Models for Robust Out-of-domain Few-Shot Learning

Namyeong Kwon, Hwidong Na+2 more|arXiv (Cornell University)|Mar 16, 2021

Domain Adaptation and Few-Shot Learning29 references6 citations

TL;DR

This paper proposes a method to repurpose pretrained MAML checkpoints for out-of-domain few-shot learning without access to the original meta-training data. By leveraging uncertainty estimates from deep ensembles to adaptively scale learning rates and generate task-specific adversarial examples, the approach improves accuracy and robustness across same-domain and cross-domain benchmarks using both SGD and Adam optimizers.

ABSTRACT

Model-agnostic meta-learning (MAML) is a popular method for few-shot learning but assumes that we have access to the meta-training set. In practice, training on the meta-training set may not always be an option due to data privacy concerns, intellectual property issues, or merely lack of computing resources. In this paper, we consider the novel problem of repurposing pretrained MAML checkpoints to solve new few-shot classification tasks. Because of the potential distribution mismatch, the original MAML steps may no longer be optimal. Therefore we propose an alternative meta-testing procedure and combine MAML gradient steps with adversarial training and uncertainty-based stepsize adaptation. Our method outperforms "vanilla" MAML on same-domain and cross-domains benchmarks using both SGD and Adam optimizers and shows improved robustness to the choice of base stepsize.

Motivation & Objective

• Address the challenge of deploying few-shot models in real-world scenarios where meta-training data is inaccessible due to privacy, licensing, or resource constraints.
• Overcome the distribution shift between meta-training and meta-testing tasks that undermines standard MAML's performance in out-of-domain settings.
• Improve robustness to hyperparameter choices—particularly base learning rate—during meta-testing when retraining is not feasible.
• Develop a meta-testing procedure that adapts MAML beyond simple gradient steps by incorporating uncertainty and adversarial data augmentation.
• Demonstrate that uncertainty-aware adaptation and adversarial training can enhance generalization in few-shot learning without retraining on the original meta-dataset.

Proposed method

• Use deep ensembles on the support set to estimate model parameter uncertainty and input gradient uncertainty during meta-testing.
• Adapt the MAML learning rate per parameter by scaling it inversely with parameter uncertainty, reducing updates on high-uncertainty components.
• Generate task-specific adversarial examples using the uncertainty of input gradients to improve robustness during adaptation.
• Apply uncertainty-guided adversarial training (UFGSM) that increases perturbation on inputs with high gradient uncertainty.
• Combine uncertainty-based stepsize adaptation and adversarial data augmentation into a modified meta-testing procedure distinct from standard MAML.
• Freeze batch normalization layers during meta-testing to validate the hypothesis that high-uncertainty layers are sensitive to large updates.

Experimental results

Research questions

• RQ1 Can uncertainty estimates from deep ensembles improve meta-testing performance when retraining on meta-training data is not possible?
• RQ2 Does uncertainty-based adaptive learning rate scheduling enhance robustness to base learning rate selection in few-shot learning?
• RQ3 Can task-specific adversarial examples generated from input gradient uncertainty improve generalization on out-of-domain few-shot tasks?
• RQ4 How does the proposed method compare to vanilla MAML in terms of accuracy and robustness across different optimizers (SGD and Adam) and datasets?
• RQ5 Is freezing high-uncertainty layers such as batch normalization beneficial during meta-testing for improving stability and performance?

Key findings

• The proposed method outperforms vanilla MAML on both same-domain and cross-domain few-shot benchmarks, achieving higher accuracy across all metrics.
• On the 5-way 1-shot MiniImageNet benchmark, the method achieved 38.71% accuracy with SGD+USA+UFGSM, compared to 36.97% with standard FGSM, demonstrating the benefit of uncertainty-based adversarial training.
• With Adam optimizer, the method achieved 44.58% top-1 accuracy on Flowers, outperforming the baseline (40.33%) and showing improved robustness across optimizers.
• Freezing batch normalization layers during meta-testing improved performance on the All metric, especially at higher base learning rates (>0.1), confirming the risk of updating high-uncertainty components.
• The method showed consistent improvements in robustness to base learning rate selection, with performance stability across a wide range of learning rates.
• Empirical ablation confirmed that using large stepsizes on high-uncertainty parameters or reducing adversarial perturbation on high-uncertainty gradients leads to lower accuracy and reduced robustness.