[Paper Review] COVID-19 Prognosis via Self-Supervised Representation Learning and Multi-Image Prediction

Anuroop Sriram, Matthew J. Muckley+8 more|PubMed|Jan 13, 2021

COVID-19 diagnosis using AIMedicine31 citations

TL;DR

The paper pretrains chest X-ray representations with MoCo self-supervised learning on public datasets and finetunes for COVID-19 deterioration prediction, including single-image and multi-image (Transformer-based) approaches, achieving higher AUC than supervised baselines and comparing to radiologists.

ABSTRACT

The rapid spread of COVID-19 cases in recent months has strained hospital resources, making rapid and accurate triage of patients presenting to emergency departments a necessity. Machine learning techniques using clinical data such as chest X-rays have been used to predict which patients are most at risk of deterioration. We consider the task of predicting two types of patient deterioration based on chest X-rays: adverse event deterioration (i.e., transfer to the intensive care unit, intubation, or mortality) and increased oxygen requirements beyond 6 L per day. Due to the relative scarcity of COVID-19 patient data, existing solutions leverage supervised pretraining on related non-COVID images, but this is limited by the differences between the pretraining data and the target COVID-19 patient data. In this paper, we use self-supervised learning based on the momentum contrast (MoCo) method in the pretraining phase to learn more general image representations to use for downstream tasks. We present three results. The first is deterioration prediction from a single image, where our model achieves an area under receiver operating characteristic curve (AUC) of 0.742 for predicting an adverse event within 96 hours (compared to 0.703 with supervised pretraining) and an AUC of 0.765 for predicting oxygen requirements greater than 6 L a day at 24 hours (compared to 0.749 with supervised pretraining). We then propose a new transformer-based architecture that can process sequences of multiple images for prediction and show that this model can achieve an improved AUC of 0.786 for predicting an adverse event at 96 hours and an AUC of 0.848 for predicting mortalities at 96 hours. A small pilot clinical study suggested that the prediction accuracy of our model is comparable to that of experienced radiologists analyzing the same information.

Motivation & Objective

• Motivate rapid, accurate triage of COVID-19 patients using chest X-ray data in resource-constrained ward settings.
• Investigate self-supervised pretraining to learn generalizable image representations before fine-tuning on COVID-19 deterioration tasks.
• Evaluate single-image and sequence-based (multi-image) prediction architectures for adverse events and oxygen requirements.
• Assess the performance of models against radiologist predictions in a pilot reader study.

Proposed method

• Pretrain DenseNet-121 encoders with Momentum Contrast (MoCo) on large public chest X-ray datasets (MIMIC-CXR-JPG and CheXpert) without using COVID labels.
• Fine-tune the pretrained encoder for downstream tasks: Single Image Prediction (SIP) of any adverse event within 24–96 hours, Oxygen Requirement Prediction (ORP) (>6 L/day) within 24–96 hours.
• Develop a Transformer-based architecture (MIP) that takes sequences of chest X-rays with relative times and uses Continuous Positional Embedding (CPE) and a time-aware transformer to predict adverse events.
• Introduce DropImage regularizer to improve robustness by randomly dropping some past images in sequences.
• Compare self-supervised pretraining with supervised pretraining and perform bootstrap-based statistical significance testing.

Experimental results

Research questions

• RQ1 Can self-supervised pretraining on non-COVID chest X-rays produce representations that improve COVID-19 deterioration prediction compared with supervised pretraining?
• RQ2 Does incorporating temporal sequences of chest X-rays via a Transformer improve prediction of adverse events and mortality compared with single-image models?
• RQ3 How does the MoCo-based approach compare to a COVID-GMIC model and to radiologists in a reader study?
• RQ4 What are the relative benefits of SIP versus ORP tasks across different prediction horizons?
• RQ5 Is there a benefit to combining public pretraining data (MIMIC-CXR, CheXpert) for pretraining when the target task involves COVID-19 patients?

Key findings

• Self-supervised MoCo pretraining yields higher AUC than supervised pretraining for SIP across all time horizons (significant at 72 and 96 hours).
• For SIP, MoCo pretraining with full fine-tuning achieves the best AUC at 48, 72, and 96 hours in several ablations, while MoCo with classifier-only fine-tuning excels at 24 hours.
• The MIP transformer model using sequences of X-rays improves AUC over single-image MoCo baselines for predicting adverse events at all time windows, with significant gains at 24, 48, and 72 hours.
• Compared to radiologists in a pilot reader study, the MIP transformer matches or slightly exceeds radiologist performance for predicting any adverse event, with radiologists showing wider variability in some tasks.
• SIP AUCs with MoCo on public data show 0.686, 0.719, 0.726, 0.742 across 24, 48, 72, 96 hours respectively (best MoCo), versus COVID-GMIC at 0.695, 0.716, 0.717, 0.738 at the same horizons.
• ORP results suggest supervised pretraining can be competitive with MoCo, with MoCo improving at 24 hours in some ablations; overall time dynamics differ between SIP and ORP tasks.
• The transformer-based MIP model with DropImage regularization provides statistically significant gains over single-image MoCo baselines in predicting ICU transfer, mortality, and overall adverse events.