[Paper Review] MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification

Jiancheng Yang, Rui Shi+6 more|arXiv (Cornell University)|Oct 27, 2021

AI in cancer detectionComputer Science35 references73 citations

TL;DR

MedMNIST v2 provides a large MNIST-like collection of standardized 2D and 3D biomedical images (12 2D datasets and 6 3D datasets) at small resolutions (28×28 / 28×28×28) with train/validation/test splits and baseline AutoML benchmarks to evaluate generalizability of ML models on medical imaging tasks.

ABSTRACT

We introduce MedMNIST v2, a large-scale MNIST-like dataset collection of standardized biomedical images, including 12 datasets for 2D and 6 datasets for 3D. All images are pre-processed into a small size of 28x28 (2D) or 28x28x28 (3D) with the corresponding classification labels so that no background knowledge is required for users. Covering primary data modalities in biomedical images, MedMNIST v2 is designed to perform classification on lightweight 2D and 3D images with various dataset scales (from 100 to 100,000) and diverse tasks (binary/multi-class, ordinal regression, and multi-label). The resulting dataset, consisting of 708,069 2D images and 10,214 3D images in total, could support numerous research / educational purposes in biomedical image analysis, computer vision, and machine learning. We benchmark several baseline methods on MedMNIST v2, including 2D / 3D neural networks and open-source / commercial AutoML tools. The data and code are publicly available at https://medmnist.com/.

Motivation & Objective

• Create a diverse, standardized, lightweight benchmark for 2D and 3D biomedical image classification to assess generalization of ML models across modalities and scales.
• Provide MNIST-like pre-processed data (28×28 for 2D; 28×28×28 for 3D) with fixed train/validation/test splits and permissive licenses for educational use.
• Enable systematic evaluation using traditional deep nets, open-source AutoML, and commercial AutoML tools, isolating machine learning aspects from end-to-end system tuning.
• Offer educational access and reproducibility by providing public data, code, and reproducible baselines for biomedical image classification.

Proposed method

• Assemble 12 2D and 6 3D pre-processed datasets in MNIST-like format (28×28 or 28×28×28) from diverse biomedical modalities.
• Standardize datasets with fixed train/validation/test splits using source-derived or partitioned splits to minimize data leakage.
• Provide data in NumPy npz format with fixed keys for images and labels (train/val/test) to streamline experimentation.
• Benchmark baseline models including ResNet variants (2D and 3D/ACS/2.5D) and autoML tools (auto-sklearn, AutoKeras, Google AutoML Vision) across all datasets.
• Evaluate using AUC and ACC to enable threshold-free and discrete performance assessment, reporting mean results over trials.
• Discuss dataset-specific idiosyncrasies (e.g., 2D vs 3D comparisons, resolution effects) and cross-dataset generalization tendencies.

Experimental results

Research questions

• RQ1 How do different machine learning models (including CNN backbones and AutoML tools) perform across a diverse set of small, standardized 2D and 3D biomedical image tasks?
• RQ2 What is the impact of input resolution (28 vs 224 in 2D) and dimensionality (2D vs 3D vs 2.5D/ACS) on classification performance?
• RQ3 Can a lightweight MNIST-like benchmark reliably reflect generalization potential of models in medical imaging when compared to end-to-end systems?
• RQ4 How do open-source versus commercial AutoML tools compare to handcrafted baselines on a broad biomedical image classification benchmark?

Key findings

• Google AutoML Vision often achieves strong average performance on 2D datasets, but does not universally outperform ResNet baselines.
• Under the same backbone, higher-resolution inputs (224) generally improve AUC/ACC over 28 for 2D datasets, while 2D ResNets can outperform deeper variants on smaller resolutions.
• In 3D, 3D convs (ResNet-50 with 3D, ACS) typically outperform 2.5D, with standard 3D convolution models delivering top-average performance among tested methods.
• Auto-sklearn can outperform some CNN baselines on certain 3D datasets but generally lags on 2D tasks; AutoKeras performs variably depending on dataset scale.
• On average across 3D datasets, 3D ResNet backbones tend to yield strong performance, with 2.5D models generally underperforming compared to full 3D methods.