[Paper Review] Assaying Out-Of-Distribution Generalization in Transfer Learning

Florian Wenzel, Andrea Dittadi+10 more|arXiv (Cornell University)|Jul 19, 2022

Domain Adaptation and Few-Shot LearningComputer Science117 references26 citations

TL;DR

The paper conducts a large-scale, unified study of OOD robustness in transfer learning, evaluating how architecture, fine-tuning, and augmentation influence OOD performance across 172 ID–OOD dataset pairs, 31k models, and 9 architectures.

ABSTRACT

Since out-of-distribution generalization is a generally ill-posed problem, various proxy targets (e.g., calibration, adversarial robustness, algorithmic corruptions, invariance across shifts) were studied across different research programs resulting in different recommendations. While sharing the same aspirational goal, these approaches have never been tested under the same experimental conditions on real data. In this paper, we take a unified view of previous work, highlighting message discrepancies that we address empirically, and providing recommendations on how to measure the robustness of a model and how to improve it. To this end, we collect 172 publicly available dataset pairs for training and out-of-distribution evaluation of accuracy, calibration error, adversarial attacks, environment invariance, and synthetic corruptions. We fine-tune over 31k networks, from nine different architectures in the many- and few-shot setting. Our findings confirm that in- and out-of-distribution accuracies tend to increase jointly, but show that their relation is largely dataset-dependent, and in general more nuanced and more complex than posited by previous, smaller scale studies.

Motivation & Objective

• Motivate the need for a unified evaluation of OOD robustness in transfer learning across realistic downstream tasks.
• Systematically compare architecture types, fine-tuning strategies, and augmentation methods under a consistent experimental protocol.
• Quantify the relationship between in-distribution accuracy and out-of-distribution performance across diverse shifts.
• Assess the informativeness of common robustness metrics beyond ID accuracy in predicting OOD robustness.

Proposed method

• Assemble 172 (ID, OOD) dataset pairs from 36 datasets spanning 10 tasks with shared labels.
• Fine-tune 31k models across 9 architectures using a fixed downstream transfer protocol and single-GPU training.
• Evaluate ID, OOD, and corrupted data using six metrics including accuracy, NLL, calibration error, and adversarial robustness.
• Compare two fine-tuning strategies (full model vs. head-only) and three augmentation regimes (no augmentation, RandAugment, AugMix).
• Analyze the transferability of metrics from ID to OOD data and the dependence of results on dataset/task shifts.

Experimental results

Research questions

• RQ1 What are good proxy measures of OOD robustness when only a single dataset is available?
• RQ2 How do architecture choices and fine-tuning strategies affect robustness to distribution shifts in transfer learning?
• RQ3 Do common robustness metrics beyond ID accuracy provide meaningful information about OOD performance?
• RQ4 Is there a consistent linear relation between ID and OOD accuracy across tasks, or are results task-dependent?
• RQ5 How well do upstream (ImageNet) robustness findings translate to downstream OOD robustness?

Key findings

• ID accuracy is the strongest predictor of OOD robustness across tasks or shifts.
• Evaluating accuracy on held-out OOD data provides an even stronger prediction of downstream OOD performance.
• Calibration and invariance metrics on ID data transfer poorly to OOD settings without OOD-heldout data, whereas invariance measures can relate to calibration under certain conditions.
• Synthetic corruptions offer limited predictive value for natural distribution shifts when adjusting for ID accuracy, though corrupted calibration can inform OOD calibration.
• Augmentations improve both ID and OOD performance, especially in low-data regimes; full fine-tuning generally outperforms head-only fine-tuning on full data, while head-only can be advantageous with scarce data.
• Architectural choices (e.g., ViT variants vs. DeiT/Swin) impact OOD performance; models optimized for transfer can differ in ID vs. OOD gaps, indicating architecture matters for robustness.