[Paper Review] Effect of large-scale pre-training on full and few-shot transfer learning for natural and medical images.

Mehdi Cherti, Jenia Jitsev|arXiv (Cornell University)|May 31, 2021

COVID-19 diagnosis using AI31 references6 citations

TL;DR

This study investigates how large-scale pre-training affects transfer learning in natural and medical imaging, comparing models pre-trained on ImageNet-21k (natural images) and large chest X-ray datasets. It finds that scaling pre-training benefits transfer most when source and target domains are similar, but these gains diminish when domains are distant—surprisingly, a large ImageNet-21k model outperforms domain-specific X-ray models in full-shot transfer to PadChest, suggesting generic data can enable high-performance domain adaptation without domain-specific pre-training data.

ABSTRACT

Transfer learning aims to exploit pre-trained models for more efficient follow-up training on wide range of downstream tasks and datasets, enabling successful training also on small data. Recent line of work posits strong benefits for model generalization and transfer when model size, data size, and compute budget are increased for the pre-training. It remains however still largely unclear whether the observed transfer improvement due to increase in scale also holds when source and target data distributions are far apart from each other. In this work we conduct large-scale pre-training on large source datasets of either natural (ImageNet-21k/1k) or medical chest X-Ray images and compare full and few-shot transfer using different target datasets from both natural and medical imaging domains. Our observations provide evidence that while pre-training and transfer on closely related datasets do show clear benefit of increasing model and data size during pre-training, such benefits are not clearly visible when source and target datasets are further apart. These observations hold across both full and few-shot transfer and indicate that scaling laws pointing to improvement of generalization and transfer with increasing model and data size are incomplete and should be revised by taking into account the type and proximity of the source and target data, to correctly predict the effect of model and data scale during pre-training on transfer. Remarkably, in full shot transfer to a large X-Ray chest imaging target (PadChest), the largest model pre-trained on ImageNet-21k slightly outperforms best models pre-trained on large X-Ray chest imaging data. This indicates possibility to obtain high quality models for domain-specific transfer even without access to large domain-specific data, by pre-training instead on comparably very large, generic source data.

Motivation & Objective

• To evaluate whether scaling pre-training (model size, data size, compute) improves transfer performance across natural and medical imaging domains.
• To assess whether benefits of large-scale pre-training hold when source and target data distributions are distant.
• To compare full and few-shot transfer performance between models pre-trained on natural (ImageNet-21k/1k) and medical (chest X-ray) data.
• To determine whether large-scale pre-training on generic data can outperform domain-specific pre-training in medical imaging.

Proposed method

• Conducted large-scale pre-training on ImageNet-21k (natural images) and large chest X-ray datasets using models of increasing size.
• Performed full and few-shot transfer learning to diverse downstream datasets from both natural and medical imaging domains.
• Evaluated performance across multiple target datasets using standard metrics, comparing models pre-trained on generic and domain-specific data.
• Systematically varied model size and data scale during pre-training to isolate the effect of scale on transfer performance.
• Used standardized training and evaluation protocols to ensure fair comparison between models pre-trained on different source data.

Experimental results

Research questions

• RQ1 Does increasing model and data scale during pre-training consistently improve transfer performance across diverse source-target domain pairs?
• RQ2 Are the benefits of large-scale pre-training diminished when source and target data distributions are far apart?
• RQ3 Can a model pre-trained on large-scale generic data (e.g., ImageNet-21k) outperform models pre-trained on domain-specific medical imaging data in downstream medical image tasks?
• RQ4 How do full-shot and few-shot transfer performance compare when scaling pre-training across natural and medical imaging domains?
• RQ5 To what extent does domain proximity moderate the effectiveness of scaling laws in transfer learning?

Key findings

• Scaling pre-training improves transfer performance when source and target domains are closely related, such as between ImageNet and similar natural image datasets.
• The benefits of scaling pre-training are not clearly visible when source and target domains are distant, such as between natural images and medical X-rays.
• In full-shot transfer to the PadChest X-ray dataset, the largest model pre-trained on ImageNet-21k slightly outperformed the best models pre-trained on large chest X-ray data.
• Few-shot transfer results showed similar trends, indicating that domain proximity, not just scale, determines transfer effectiveness.
• The findings suggest that current scaling laws for transfer learning are incomplete and must incorporate data domain proximity to predict performance gains accurately.