[Paper Review] The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning

Anders Andreassen, Yasaman Bahri+2 more|arXiv (Cornell University)|Jun 30, 2021

Adversarial Robustness in Machine LearningComputer Science48 references21 citations

TL;DR

This paper investigates the evolution of out-of-distribution (OOD) robustness during fine-tuning, revealing that pre-trained models exhibit transient effective robustness (ER) that peaks mid-training and vanishes by convergence. Despite increasing in-distribution accuracy, these models lose their ability to generalize to OOD data, highlighting a critical trade-off in fine-tuning that current methods fail to resolve.

ABSTRACT

Although machine learning models typically experience a drop in performance on out-of-distribution data, accuracies on in- versus out-of-distribution data are widely observed to follow a single linear trend when evaluated across a testbed of models. Models that are more accurate on the out-of-distribution data relative to this baseline exhibit "effective robustness" and are exceedingly rare. Identifying such models, and understanding their properties, is key to improving out-of-distribution performance. We conduct a thorough empirical investigation of effective robustness during fine-tuning and surprisingly find that models pre-trained on larger datasets exhibit effective robustness during training that vanishes at convergence. We study how properties of the data influence effective robustness, and we show that it increases with the larger size, more diversity, and higher example difficulty of the dataset. We also find that models that display effective robustness are able to correctly classify 10% of the examples that no other current testbed model gets correct. Finally, we discuss several strategies for scaling effective robustness to the high-accuracy regime to improve the out-of-distribution accuracy of state-of-the-art models.

Motivation & Objective

• To understand the dynamics of effective robustness (ER) throughout the fine-tuning process.
• To identify why pre-trained models show transient ER while randomly initialized models do not.
• To investigate how data properties—such as size, diversity, and example difficulty—influence ER.
• To determine whether strategies like replay buffers or weight regularization can sustain high ER at high in-distribution accuracy.
• To explore the potential of scaling ER to achieve state-of-the-art OOD performance.

Proposed method

• Empirically track ER across checkpoints during fine-tuning of models on ImageNet and CIFAR-10.
• Define ER as the deviation of OOD accuracy from the linear ID-OOD performance trend observed across models.
• Use multi-label mapping between CIFAR-10 and ImageNet to enable fine-tuning with shared prediction heads.
• Apply replay buffers with ImageNet data during CIFAR-10 fine-tuning to simulate continual learning.
• Introduce L2 regularization to preserve pre-trained head weights and assess its impact on ER.
• Analyze the shape of the ER curve and decompose it into contributions from the linear fit and actual deviation.

Experimental results

Research questions

• RQ1 Why do pre-trained models exhibit effective robustness during fine-tuning, but lose it by convergence?
• RQ2 How do data properties—size, diversity, and example difficulty—affect the emergence of effective robustness?
• RQ3 Can strategies like replay buffers or weight regularization maintain high effective robustness at high in-distribution accuracy?
• RQ4 What distinguishes the predictions of effectively robust models from standard models?
• RQ5 Why is it so rare for models to maintain high ER while achieving high in-distribution accuracy?

Key findings

• Pre-trained models exhibit a transient peak in effective robustness during fine-tuning, which diminishes to zero by convergence, despite increasing in-distribution accuracy.
• The peak of effective robustness occurs when models are moderately accurate, not at convergence, indicating a trade-off between ID and OOD performance.
• Models with effective robustness correctly classify 10% of examples that no other model in the testbed gets right, demonstrating unique generalization capacity.
• Effective robustness increases with larger, more diverse, and more difficult training datasets, suggesting data quality and variety are key.
• Replay buffers and L2 regularization around pre-trained weights fail to sustain high effective robustness, even when they preserve high in-distribution accuracy.
• The shape of the ER curve is partly an artifact of the linear fit in logit space, but the actual deviation from the fit still decreases during training, indicating a real loss of OOD generalization.