[Paper Review] EvalBlocks: A Modular Pipeline for Rapidly Evaluating Foundation Models in Medical Imaging

Jan Tagscherer, Sarah de Boer+8 more|arXiv (Cornell University)|Jan 7, 2026

Radiomics and Machine Learning in Medical Imaging0 citations

TL;DR

EvalBlocks provides a Snakemake-based, modular workflow to rapidly evaluate multiple foundation models on medical imaging tasks with caching, reproducibility, and cluster-ready execution.

ABSTRACT

Developing foundation models in medical imaging requires continuous monitoring of downstream performance. Researchers are burdened with tracking numerous experiments, design choices, and their effects on performance, often relying on ad-hoc, manual workflows that are inherently slow and error-prone. We introduce EvalBlocks, a modular, plug-and-play framework for efficient evaluation of foundation models during development. Built on Snakemake, EvalBlocks supports seamless integration of new datasets, foundation models, aggregation methods, and evaluation strategies. All experiments and results are tracked centrally and are reproducible with a single command, while efficient caching and parallel execution enable scalable use on shared compute infrastructure. Demonstrated on five state-of-the-art foundation models and three medical imaging classification tasks, EvalBlocks streamlines model evaluation, enabling researchers to iterate faster and focus on model innovation rather than evaluation logistics. The framework is released as open source software at https://github.com/DIAGNijmegen/eval-blocks.

Motivation & Objective

• Motivate the need for rapid, reproducible evaluation of foundation models during medical imaging model development.
• Propose a modular, plug-and-play evaluation framework built on Snakemake.
• Demonstrate the framework by evaluating multiple foundation models on several medical imaging classification tasks.
• Highlight how centralized experiment tracking and caching accelerate iteration and reduce logistics burden.

Proposed method

• Implement a modular pipeline where blocks perform feature extraction, optional aggregation, and evaluation as Snakemake rules.
• Support seamless integration of new datasets, foundation models, aggregation methods, and evaluation strategies.
• Use centralized configuration to declare experiments and enable reproducible results with a single command.
• Enable production-like caching and parallel execution suitable for cluster environments (e.g., Slurm).
• Evaluate five foundation models across three malignant classification tasks using multiple aggregation and evaluation strategies.
• Provide visualization blocks for embedding analyses (LDA, PCA, t-SNE) to aid interpretation.

Figure 1 : In our framework, pipeline steps are implemented as self-contained blocks. Foundation models embed input patches, and these feature embeddings can be optionally aggregated and then evaluated. The pipeline blocks can be freely extended and plugged into each other, enabling fast, reproducib

Experimental results

Research questions

• RQ1 How can a modular, reusable pipeline accelerate iterative evaluation of foundation models in medical imaging?
• RQ2 What is the impact of different aggregation strategies on downstream classification performance across models and modalities?
• RQ3 Can centralized experiment tracking and caching reduce evaluation time and improve reproducibility when comparing multiple foundation models?
• RQ4 How do various evaluation strategies (kNN, linear probing, visualization) compare in discriminating malignancies across datasets?
• RQ5 What is the practicality of running such a framework on local and cluster infrastructure for in-house datasets?

Key findings

• EvalBlocks enables automated, reproducible evaluation across multiple models, tasks, and configurations.
• The framework supports fast iteration through caching and parallel execution, reducing wall-time.
• Different models show varying strengths depending on dataset and modality, with no single model dominating all tasks in the demonstration.
• Aggregation methods (e.g., mean across modalities) can influence performance and facilitate rapid prototyping of fusion strategies.
• Visualization analyses provide interpretable insights into embedding separability and guide further model development.

Figure 2 : A visualization of model results on our CT datasets created by running EvalBlocks, with error bars depicting the standard deviation across folds. While CT-FM [ 9 ] and Curia [ 3 ] perform best on PANORAMA [ 1 ] , UMedPT [ 11 ] is slightly more accurate on AMARA. Our pipeline allows for fa