[Paper Review] Transfusion: Understanding Transfer Learning for Medical Imaging

Maithra Raghu, Chiyuan Zhang+2 more|arXiv (Cornell University)|Feb 14, 2019

COVID-19 diagnosis using AI40 references642 citations

TL;DR

Transfer learning from ImageNet provides limited gains for two large medical imaging tasks; smaller lightweight models perform comparably, and substantial feature reuse is confined to the lowest layers with additional weight-scaling benefits aiding convergence.

ABSTRACT

Transfer learning from natural image datasets, particularly ImageNet, using standard large models and corresponding pretrained weights has become a de-facto method for deep learning applications to medical imaging. However, there are fundamental differences in data sizes, features and task specifications between natural image classification and the target medical tasks, and there is little understanding of the effects of transfer. In this paper, we explore properties of transfer learning for medical imaging. A performance evaluation on two large scale medical imaging tasks shows that surprisingly, transfer offers little benefit to performance, and simple, lightweight models can perform comparably to ImageNet architectures. Investigating the learned representations and features, we find that some of the differences from transfer learning are due to the over-parametrization of standard models rather than sophisticated feature reuse. We isolate where useful feature reuse occurs, and outline the implications for more efficient model exploration. We also explore feature independent benefits of transfer arising from weight scalings.

Motivation & Objective

• Motivate and evaluate the effectiveness of transfer learning from natural images to medical imaging tasks.
• Compare standard ImageNet architectures with smaller lightweight models on two large medical datasets.
• Analyze learned representations to understand feature reuse and identify where transfer helps.
• Investigate convergence benefits from pretrained weights independent of feature reuse.
• Propose hybrid transfer strategies that balance performance and computational efficiency.

Proposed method

• Evaluate multiple architectures (ResNet-50, Inception-v3, and a family of lightweight CNNs called CBR) with random initialization and with ImageNet pretraining.
• Use two large medical datasets (Retina fundus images and CheXpert chest X-rays) and measure AUC-ROC across tasks.
• Analyze hidden representations with SVCCA to assess representational similarity before and after fine-tuning.
• Perform weight transfusion experiments by reusing subsets of pretrained weights and redesigning the top of the network.
• Examine feature-independent benefits of transfer via weight scaling (Mean Var initialization) to study convergence speed.
• Visualize early-layer filters to understand how pretrained features adapt during training.

Experimental results

Research questions

• RQ1 Does transfer learning from ImageNet improve medical imaging task performance relative to random initialization?
• RQ2 Are lightweight architectures able to match or surpass ImageNet-class architectures on medical tasks?
• RQ3 Where in the network does feature reuse from pretraining occur in medical imaging models?
• RQ4 Do pretrained weights impact convergence speed independently of feature reuse?
• RQ5 Can hybrid transfer strategies maintain benefits while enabling more flexible model design?

Key findings

• Transfer learning offers limited performance gains across both medical tasks and architectures.
• Smaller, simpler CNNs (CBR) achieve comparable performance to standard ImageNet models on Retina and CheXpert tasks.
• ImageNet top-5 accuracy is not predictive of medical task performance.
• Representational analysis shows that large models change less during training and feature reuse is mainly limited to the lowest layers.
• Feature-independent benefits of transfer exist via weight scaling that speeds up convergence (Mean Var initialization).
• Hybrid approaches (reusing only the lowest layers or combining with lightweight top redesigns, or using synthetic conv1 features) can match full transfer learning performance with added flexibility.