[Paper Review] Towards Foundation Models and Few-Shot Parameter-Efficient Fine-Tuning for Volumetric Organ Segmentation

Julio Silva-Rodríguez, José Dolz+1 more|arXiv (Cornell University)|Mar 29, 2023

Radiomics and Machine Learning in Medical ImagingMedicine3 citations

TL;DR

This paper introduces a novel few-shot efficient fine-tuning (FSEFT) framework for volumetric organ segmentation using foundation models, proposing spatial adapter modules and constrained transductive inference to enable parameter-efficient adaptation with minimal labeled data. The method achieves performance close to full-supervision fine-tuning while reducing parameter updates and improving robustness in low-data clinical settings.

ABSTRACT

The recent popularity of foundation models and the pre-train-and-adapt paradigm, where a large-scale model is transferred to downstream tasks, is gaining attention for volumetric medical image segmentation. However, current transfer learning strategies devoted to full fine-tuning for transfer learning may require significant resources and yield sub-optimal results when the labeled data of the target task is scarce. This makes its applicability in real clinical settings challenging since these institutions are usually constrained on data and computational resources to develop proprietary solutions. To address this challenge, we formalize Few-Shot Efficient Fine-Tuning (FSEFT), a novel and realistic scenario for adapting medical image segmentation foundation models. This setting considers the key role of both data- and parameter-efficiency during adaptation. Building on a foundation model pre-trained on open-access CT organ segmentation sources, we propose leveraging Parameter-Efficient Fine-Tuning and black-box Adapters to address such challenges. Furthermore, novel efficient adaptation methodologies are introduced in this work, which include Spatial black-box Adapters that are more appropriate for dense prediction tasks and constrained transductive inference, leveraging task-specific prior knowledge. Our comprehensive transfer learning experiments confirm the suitability of foundation models in medical image segmentation and unveil the limitations of popular fine-tuning strategies in few-shot scenarios.

Motivation & Objective

• To formalize a realistic few-shot fine-tuning setting for medical image segmentation, where labeled data is scarce due to clinical constraints.
• To address the limitations of standard fine-tuning in low-data regimes, including overfitting and high computational costs.
• To develop a parameter-efficient fine-tuning strategy tailored for 3D volumetric segmentation, leveraging spatial adapters and task-specific inductive biases.
• To evaluate the effectiveness of the proposed method across diverse public CT datasets under few-shot conditions.

Proposed method

• Introduces spatial adapter modules inserted between layers of a pre-trained 3D segmentation backbone, specifically designed for dense prediction tasks like organ segmentation.
• Employs a constrained transductive inference mechanism that incorporates anatomical priors (e.g., organ proportion constraints) during inference to improve generalization with few shots.
• Uses a pretrain-and-adapt paradigm with a foundation model pre-trained on the BTCV dataset, fine-tuned via only the adapter parameters.
• Applies a hybrid training-inference strategy where the adapter is trained on few labeled samples, and inference is regularized using prior knowledge about organ morphology.
• Optimizes only the adapter parameters while keeping the main backbone weights frozen, significantly reducing memory and compute requirements.
• Employs a multi-dataset evaluation protocol across public CT datasets to assess zero-shot and few-shot transferability.

Experimental results

Research questions

• RQ1 Can standard fine-tuning methods generalize effectively when only a few labeled samples are available for new organ segmentation tasks?
• RQ2 How do parameter-efficient fine-tuning strategies compare to full fine-tuning in low-data scenarios for 3D medical image segmentation?
• RQ3 To what extent do spatial adapters improve performance on dense prediction tasks compared to standard adapter modules?
• RQ4 Does incorporating anatomical priors via constrained transductive inference enhance segmentation accuracy in few-shot settings?
• RQ5 Can foundation models pre-trained on diverse datasets be effectively adapted to new clinical tasks with minimal labeled data?

Key findings

• Standard fine-tuning methods suffer significant performance drops in few-shot scenarios, with Dice scores dropping below 0.5 in some cases when using only 1–5 labeled samples.
• The proposed spatial adapter module improves mean Dice score by up to 0.012 over standard adapters, particularly benefiting organs with complex shapes like the liver and pancreas.
• Constrained transductive inference further boosts performance, increasing the Dice score by up to 0.016 in 5-shot settings, especially on small or irregularly shaped organs.
• The full framework (adapter + transductive inference) achieves a mean Dice score of 0.783 on the 10-shot setting, approaching the performance of full fine-tuning (0.787) while updating only 0.04% of parameters.
• Linear probing and standard adapters underperform in low-data regimes, with linear probing achieving only 0.771 mean Dice in 10-shot, highlighting the need for more expressive adaptation mechanisms.
• The method generalizes well across diverse CT datasets, with consistent improvements over strong baselines even when adapting to new organs and imaging protocols.