[Paper Review] Less Could Be Better: Parameter-efficient Fine-tuning Advances Medical Vision Foundation Models

Chenyu Lian, Hongyu Zhou+2 more|arXiv (Cornell University)|Jan 22, 2024

COVID-19 diagnosis using AI5 citations

TL;DR

The paper evaluates LoRA, a parameter-efficient fine-tuning method, against full-parameter fine-tuning on two self-supervised chest radiography foundation models across three datasets, showing LoRA often beats FFT with far fewer tunable parameters and achieves state-of-the-art data efficiency.

ABSTRACT

Parameter-efficient fine-tuning (PEFT) that was initially developed for exploiting pre-trained large language models has recently emerged as an effective approach to perform transfer learning on computer vision tasks. However, the effectiveness of PEFT on medical vision foundation models is still unclear and remains to be explored. As a proof of concept, we conducted a detailed empirical study on applying PEFT to chest radiography foundation models. Specifically, we delved into LoRA, a representative PEFT method, and compared it against full-parameter fine-tuning (FFT) on two self-supervised radiography foundation models across three well-established chest radiograph datasets. Our results showed that LoRA outperformed FFT in 13 out of 18 transfer learning tasks by at most 2.9% using fewer than 1% tunable parameters. Combining LoRA with foundation models, we set up new state-of-the-art on a range of data-efficient learning tasks, such as an AUROC score of 80.6% using 1% labeled data on NIH ChestX-ray14. We hope this study can evoke more attention from the community in the use of PEFT for transfer learning on medical imaging tasks. Code and models are available at https://github.com/RL4M/MED-PEFT.

Motivation & Objective

• Motivate exploring parameter-efficient fine-tuning (PEFT) for medical vision foundation models due to limited annotations in medical imaging.
• Assess LoRA, a representative PEFT method, against full-parameter fine-tuning on chest radiography foundation models.
• Evaluate transfer learning performance across NIH ChestX-ray14, CheXpert, and RSNA pneumonia datasets.
• Analyze data efficiency and scalability of LoRA with different model sizes and pre-training setups.

Proposed method

• Compare LoRA and FFT on two self-supervised radiography foundation models (MRM and MAE) pre-trained on MIMIC-CXR.
• Assess AUROC across NIH, CheXpert, RSNA under 1%, 10%, and 100% labeled data regimes.
• Evaluate impact of LoRA rank, pre-training epochs, and model scaling on performance.
• Provide ablations and analyses to understand data efficiency and robustness of LoRA in medical imaging.
• Release code and models at the provided GitHub repository.

Experimental results

Research questions

• RQ1 Does LoRA consistently outperform full fine-tuning (FFT) in medical chest radiography tasks across datasets and data regimes?
• RQ2 How data-efficient is LoRA (in terms of labeled data percentage) compared to FFT for medical vision foundation models?
• RQ3 How do model size, pre-training epochs, and LoRA rank affect performance on chest X-ray transfer tasks?
• RQ4 Can PEFT approaches like LoRA enable competitive or superior performance with minimal tunable parameters in medical imaging?

Key findings

• LoRA outperformed FFT in 13 of 18 transfer learning tasks across MAE and MRM radiography foundation models.
• LoRA achieved up to 2.9% AUROC gains over FFT while tuning less than 1% of parameters.
• LoRA demonstrated strong data efficiency, with notable gains at 1% and 10% labeled data (e.g., NIH ChestX-ray14 results).
• With 100% labeled data, LoRA matched FFT performance while tuning only about 1.5% of parameters.
• Scaling up foundation models and using LoRA on larger ViT backbones yielded additional AUROC improvements (e.g., ViT-Large with LoRA achieving higher AUROC than FFT in some settings).
• LoRA-based results on natural image pre-trained models showed the necessity of chest radiography pre-training to achieve strong downstream performance, but LoRA mitigated modality gaps compared to FFT.