[Paper Review] Estimating Uncertainty and Interpretability in Deep Learning for Coronavirus (COVID-19) Detection

Biraja Ghoshal, Allan Tucker|Brunel University Research Archive (BURA) (Brunel University London)|Mar 22, 2020

COVID-19 diagnosis using AI14 references348 citations

TL;DR

The paper uses MC Dropweights-based Bayesian CNNs to estimate predictive uncertainty in COVID-19 chest X-ray classification and shows uncertainty correlates with accuracy, enabling uncertainty-aware referrals to radiologists.

ABSTRACT

Deep Learning has achieved state of the art performance in medical imaging. However, these methods for disease detection focus exclusively on improving the accuracy of classification or predictions without quantifying uncertainty in a decision. Knowing how much confidence there is in a computer-based medical diagnosis is essential for gaining clinicians trust in the technology and therefore improve treatment. Today, the 2019 Coronavirus (SARS-CoV-2) infections are a major healthcare challenge around the world. Detecting COVID-19 in X-ray images is crucial for diagnosis, assessment and treatment. However, diagnostic uncertainty in the report is a challenging and yet inevitable task for radiologist. In this paper, we investigate how drop-weights based Bayesian Convolutional Neural Networks (BCNN) can estimate uncertainty in Deep Learning solution to improve the diagnostic performance of the human-machine team using publicly available COVID-19 chest X-ray dataset and show that the uncertainty in prediction is highly correlates with accuracy of prediction. We believe that the availability of uncertainty-aware deep learning solution will enable a wider adoption of Artificial Intelligence (AI) in a clinical setting.

Motivation & Objective

• Motivate the need for uncertainty estimation in deep learning for medical imaging COVID-19 detection.
• Evaluate MC Dropweights-based Bayesian CNNs to quantify predictive uncertainty.
• Investigate the relationship between predictive uncertainty and classification accuracy.
• Assess uncertainty-aware decision referral to improve human–machine collaboration in diagnosis.

Proposed method

• Apply transfer learning with a pre-trained ResNet50V2 backbone for COVID-19 X-ray classification.
• Incorporate Dropweights in a fully connected layer to approximate Bayesian inference and enable MC sampling during testing.
• Compute predictive mean and predictive uncertainty over multiple stochastic forward passes (MC samples) to obtain PH (predictive entropy) and BALD measures.
• Normalise uncertainty across configurations to facilitate comparisons.
• Use an asymmetric cost (utility) function to emphasize reduction of false negatives via class weights in the loss.
• Compare uncertainty measures (PH vs BALD) and analyze how uncertainty relates to prediction errors.

Experimental results

Research questions

• RQ1 Can MC Dropweights-based Bayesian CNNs provide meaningful uncertainty estimates for COVID-19 X-ray classification?
• RQ2 Is predictive uncertainty correlated with classification accuracy, and can it be used to improve decision making through referrals?
• RQ3 Which uncertainty metric (predictive entropy vs BALD) better reflects model uncertainty in this context?
• RQ4 Does incorporating uncertainty-aware referral improve overall diagnostic reliability when combined with human radiologists?

Key findings

• Bayesian ResNet50V2 improves COVID-19 X-ray detection accuracy over a standard ResNet50V2 on the given dataset.
• Predictive uncertainty tends to be higher for incorrect predictions, indicating usefulness for flagging ambiguous cases.
• Predictive entropy shows a very strong correlation with prediction error (rho ≈ 0.99 across tested dropweights).
• BALD also correlates with errors but to a lesser extent than predictive entropy across configurations.
• Uncertainty-aware referral (rejecting uncertain cases) increases accuracy of the remaining predictions, and the combined system approaches radiologist-like performance under certain conditions.