[Paper Review] A Critic Evaluation of Methods for COVID-19 Automatic Detection from X-Ray Images

Gianluca Maguolo, Loris Nanni|arXiv (Cornell University)|Apr 27, 2020

COVID-19 diagnosis using AI4 citations

TL;DR

This paper critically evaluates deep learning methods for automatic COVID-19 detection from X-ray images, demonstrating that models can achieve high accuracy even when lung regions are removed, indicating that models learn spurious patterns unrelated to disease. The authors propose a fairness metric to assess testing protocols and show that current evaluation practices are fundamentally flawed due to dataset biases.

ABSTRACT

In this paper, we compare and evaluate different testing protocols used for automatic COVID-19 diagnosis from X-Ray images in the recent literature. We show that similar results can be obtained using X-Ray images that do not contain most of the lungs. We are able to remove the lungs from the images by turning to black the center of the X-Ray scan and training our classifiers only on the outer part of the images. Hence, we deduce that several testing protocols for the recognition are not fair and that the neural networks are learning patterns in the dataset that are not correlated to the presence of COVID-19. Finally, we show that creating a fair testing protocol is a challenging task, and we provide a method to measure how fair a specific testing protocol is. In the future research we suggest to check the fairness of a testing protocol using our tools and we encourage researchers to look for better techniques than the ones that we propose.

Motivation & Objective

• To investigate the validity of current testing protocols used in deep learning models for COVID-19 detection from X-ray images.
• To assess whether models learn disease-relevant features or spurious patterns in the data.
• To demonstrate that models can achieve high performance even when lung regions are removed from input images.
• To propose a quantitative method for measuring the fairness of a testing protocol in medical image classification.
• To encourage future research to adopt more rigorous and unbiased evaluation standards.

Proposed method

• The authors trained classifiers on X-ray images where the central region—containing the lungs—was masked out and set to black.
• They evaluated model performance on standard benchmark datasets using the same training and testing protocols as in prior work.
• They introduced a fairness metric based on the model's performance when the lungs are removed, measuring how much performance degrades under such perturbation.
• The method evaluates whether the model relies on anatomical features or non-lung regions for classification.
• They applied the metric to multiple existing models and datasets to compare fairness across different protocols.
• The approach uses standard deep learning architectures (e.g., ResNet) and standard data augmentation techniques, but with controlled image masking.

Experimental results

Research questions

• RQ1 Can deep learning models for COVID-19 detection achieve high accuracy when the lungs are completely removed from X-ray images?
• RQ2 To what extent do current testing protocols in the literature rely on non-lung features rather than disease-specific patterns?
• RQ3 How can the fairness of a testing protocol in medical image classification be quantitatively measured?
• RQ4 What are the implications of using biased or non-representative data for model generalization and clinical reliability?
• RQ5 How can researchers ensure that models are learning relevant diagnostic features rather than dataset artifacts?

Key findings

• Models trained on X-ray images with the lungs removed still achieve high classification accuracy, indicating reliance on non-lung features.
• The performance of models remains high even when the central region of the X-ray is blacked out, suggesting that the models are not learning lung-specific patterns.
• The proposed fairness metric successfully identifies testing protocols that are not robust to image masking, revealing bias in current evaluation practices.
• Several existing methods in the literature produce high accuracy not because of disease detection, but due to dataset-specific artifacts or non-anatomical patterns.
• The study demonstrates that fair evaluation requires more than just accuracy; it demands validation of model robustness to structural perturbations like lung removal.
• The authors conclude that current evaluation protocols are not reliable for assessing true diagnostic capability and call for standardized fairness assessment in future work.