[Paper Review] Intra-model Variability in COVID-19 Classification Using Chest X-ray Images

Brian D. Goodwin, Corey Jaskolski+2 more|arXiv (Cornell University)|Apr 30, 2020

COVID-19 diagnosis using AI30 references9 citations

TL;DR

This study evaluates intra-model variability in COVID-19 classification using chest X-ray images across 12 deep learning architectures, training each model five times with identical data splits to isolate variability from weight initialization, data augmentation, and batch shuffling. Best models achieved a 15% false negative rate (3/20) on a held-out test set, highlighting significant performance inconsistency despite identical architectures and data, underscoring the urgent need for larger, more diverse datasets to improve clinical reliability.

ABSTRACT

X-ray and computed tomography (CT) scanning technologies for COVID-19 screening have gained significant traction in AI research since the start of the coronavirus pandemic. Despite these continuous advancements for COVID-19 screening, many concerns remain about model reliability when used in a clinical setting. Much has been published, but with limited transparency in expected model performance. We set out to address this limitation through a set of experiments to quantify baseline performance metrics and variability for COVID-19 detection in chest x-ray for 12 common deep learning architectures. Specifically, we adopted an experimental paradigm controlling for train-validation-test split and model architecture where the source of prediction variability originates from model weight initialization, random data augmentation transformations, and batch shuffling. Each model architecture was trained 5 separate times on identical train-validation-test splits of a publicly available x-ray image dataset provided by Cohen et al. (2020). Results indicate that even within model architectures, model behavior varies in a meaningful way between trained models. Best performing models achieve a false negative rate of 3 out of 20 for detecting COVID-19 in a hold-out set. While these results show promise in using AI for COVID-19 screening, they further support the urgent need for diverse medical imaging datasets for model training in a way that yields consistent prediction outcomes. It is our hope that these modeling results accelerate work in building a more robust dataset and a viable screening tool for COVID-19.

Motivation & Objective

• To quantify variability in model performance for COVID-19 detection using chest X-ray images despite identical model architectures and data splits.
• To assess the impact of random factors—weight initialization, data augmentation, and batch shuffling—on prediction reliability.
• To evaluate baseline performance metrics across 12 common deep learning architectures on a public chest X-ray dataset.
• To provide pre-trained model weights and training code to accelerate community research and improve model training efficiency.
• To highlight the limitations of current datasets and advocate for more diverse, abundant data to achieve clinically viable AI screening tools.

Proposed method

• A fixed 80-10-10 train-validation-test split was applied across all experiments to isolate variability from data partitioning.
• Each of the 12 deep learning architectures was trained five times with identical data splits but different random seeds to vary weight initialization, data augmentation, and batch shuffling.
• Experiments were conducted on Lambda Labs GPU hardware using PyTorch, with model performance evaluated on a held-out test set.
• The Cohen et al. (2020) public chest X-ray dataset, containing healthy, community-acquired pneumonia, and COVID-19 classes, was used as the primary data source.
• Performance was measured using standard metrics including accuracy, precision, recall, and false negative rate (FNR), with a focus on FNR due to clinical implications.
• Pre-trained model weights and training code were released under a Creative Commons Attribution-NonCommercial 4.0 License to support community research.

Experimental results

Research questions

• RQ1 How much variability in prediction performance exists across multiple training runs of the same deep learning architecture on the same chest X-ray dataset?
• RQ2 What are the primary sources of performance variation in COVID-19 classification models when model architecture and data split are held constant?
• RQ3 What is the best achievable false negative rate for COVID-19 detection using current deep learning models on publicly available chest X-ray data?
• RQ4 How do performance metrics vary across different deep learning architectures when trained on the same data with controlled randomness?
• RQ5 To what extent do current public datasets limit the reliability and consistency of AI models for clinical COVID-19 screening?

Key findings

• Best-performing models achieved a false negative rate of 15% (3 out of 20) on the held-out test set, indicating meaningful but suboptimal detection performance.
• Significant intra-model variability was observed even when model architecture and data splits were held constant, with performance differing meaningfully across multiple training runs.
• The study identified that weight initialization, data augmentation, and batch shuffling were major contributors to prediction variability, undermining model consistency.
• Despite high performance in some runs, the overall results indicate that current models remain unreliable for clinical deployment due to inconsistent behavior across identical architectures.
• The findings underscore that model reliability is currently constrained by dataset limitations, not architectural design, and that larger, more diverse datasets are essential for consistent outcomes.
• The authors released pre-trained weights and training code to accelerate research and improve training efficiency, aiming to support the development of more robust screening tools.